open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity
llm-ingestion-okf/src/llm_ingestion_okf/cli.py
Kjell Tore Guttormsen 46e555d460 docs(consume,build): both parent defaults stay off, and the reason is now a measurement 
K3-21 B. `cli.DEFAULT_SHELL_PARENT`'s comment said it was off because `okf
consume` reads no `parent` key; after A that sentence was false and the
default was unmeasured in both directions. Re-measured on the one standard
with heading-only sections: hit@1, hit@8 and hit@50 stay 6/6 at both k with
the known-positive at rank 1, but the link's bundle-absolute path repeats
the document's directory in 675 bodies, and the delivered set moved on 2 of
8 questions at the default k and 3 of 8 at k 50, newly delivered shells
matching the question only through that path on 1 of 3, 4 of 7 and 6 of 23
-- the saturation K3-20 took out of the id signal, back through the body.
Off.

`consume.DEFAULT_FOLLOW_PARENT`: the same delivered set as the pointer (16
of 16 payloads), 6 of 6 correct either way on questions whose answers are
never shells, at +2 488 / +2 746 B on the 2 of 8 default-k payloads holding a
linked shell and up to +20 817 B at k 50. Off.

A test holds both defaults and that the false sentence is gone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 13:39:50 +02:00

1048 lines
47 KiB
Python
Raw Blame History

"""`okf` — the installed command. Five subcommands: the whole chain, packaged.
`build` a bundle from a folder, `consume` it for one question, `check` the
payload against the contract, generate a `skill` for it, or do the whole thing
in one step with `project`. Four of the five moved here on 2026-09-08 (O5)
from unpackaged scripts under `tools/`, which a consumer who installs this
library does not have; until then the reading direction existed only for
someone standing in a clone.
## What it replaces
Until this module existed, "run the door over a folder" was a shell loop over
`tools/okf_propose_segments.py` followed by `tools/okf_corpus_run.py`, with
nine flags between them and a `--path-prefix` rule that lived in a code block
in a measurement report. Neither script was packaged, so the whole path was
reachable only from a clone -- a consumer who installed this library could call
`process_inbox` but could not run the thing the reports measured.
`okf build <inbox> --bundle <dir>` is that path, packaged. It is ORCHESTRATION
and nothing else: every rule it applies belongs to `propose` or `corpus`, which
are the same modules `tools/` now calls. There is one implementation of each,
and it is the packaged one.
## The two decisions this layer owns
**The scope of a document's proposed paths is its RELATIVE PATH, minus the
extension.** Section numbering is document-local, so two documents propose the
same path and Door B refuses both; the loop in the reports passed the BASENAME,
which was right while the inbox was flat. It stopped being right when the door
started walking recursively: `a/krav.pdf` and `b/krav.pdf` would both reduce to
`krav` and collide, which is the collision Door B is supposed to make
impossible rather than merely detectable. For a flat inbox the relative path IS
the basename, which is why the published bundles' bytes do not move.
**Omitted timestamps do not come from the clock.** `--ingested-at` defaults to
`DEFAULT_STAMP`, and `--proposed-at` defaults to `--ingested-at` -- so a caller
who names one clock stamps the whole bundle with it, and a caller who names
neither gets `DEFAULT_STAMP` for both, one constant either way. A wall-clock
default would put a changing byte into the artifact and break
rebuild-equals-incremental (K6) for every caller who did not pass the flags --
the property the segmented bundle is built on, and the one a convenience
default is most likely to take away silently. The epoch is deliberate and
readable as what it is: a stamp nobody set. A caller who wants the proposal and
the replay dated differently passes both explicitly.
## What it does not decide
**Which arm ships as the default is the operator's decision and is not taken
here.** Three of the six are ON as of 2026-09-08 -- `--outline-run 3`,
`--table-grid` and `--unit-fold` -- each moved by the operator on a
measurement and each with an explicit opt-out. The remaining three are OFF
unless the caller asks. This module's job is to state the answer, never to
pick it.
The arms are, however, REACHABLE from here, and that is a change of 2026-09-08.
Until then `_propose_plans` called the proposer with no arm flag at all, so
`okf build` ran Arm B while `tools/okf_propose_segments.py` could run Arm D, E
and F -- a build path a full arm behind the proposer, reachable only by
retyping the loop the packaging removed. Measured on a five-document folder:
one tender PDF lands as ONE concept from the build path and as NINE with
`--outline-run 3`, and nine is what the operator's unit worksheet asks for.
Arm C (`--max-segment-chars`) stays unexposed: it is a character cap whose
value nobody has measured against a reference, so it has no number to offer a
caller.
"""
from __future__ import annotations
import argparse
import sys
import tempfile
from collections.abc import Mapping, Sequence
from functools import partial
from pathlib import Path
from .corpus import LOG_NAME, CorpusReport, load_plans, measure
from .errors import IngestError
from .extract import declared_identity
from .inbox import validate_concept_frontmatter, walk_inbox
from .materialize import reduce_to_id_grammar
from .profiles import SEGMENTED_OKF_V0_2, STRUCTURED_V1, BundleProfile
from .propose import ProposerError, heading_reserve_applies
from .propose import run as propose_run
__all__ = ["DEFAULT_STAMP", "build", "main", "measure"]
#: The subcommands whose FLAGS belong to the module that implements them.
#:
#: Dispatched before `argparse` runs rather than rebuilt as subparsers here: a
#: second declaration of `--reserve-top-rank` or `--rarity-weight` is a second
#: place they can drift, and the pre-pass's flag set is the thing four
#: measurement reports are pinned to. Each entry is registered as a subparser
#: below anyway, so `okf --help` lists it and an unknown command is still an
#: error -- the parser knows the NAMES, the modules keep the flags.
#:
#: Imported lazily inside the dispatch: `okf build` should not pay to import
#: the ranker, and `okf consume` should not pay to import the proposer.
DELEGATED = ("consume", "check", "skill", "project")
def _delegate(command: str, argv: list[str]) -> int:
if command == "consume":
from .consume import main as run
elif command == "check":
from .contract_check import main as run
elif command == "skill":
from .skill import main as run
else:
from .project import main as run
return run(argv)
CLI_ID = "okf build"
#: What `okf build` runs when no flag is given. Moved 2026-09-08 by the
#: operator, from Arm B to Arm D plus Arm F, on the round-2 and round-3
#: measurements: the previous default was a full arm behind the proposer on
#: the one document with a reference, and the fold matches 5 of 12 documents
#: against the default's 2 with no cell worse.
#:
#: THE VALUE LIVES HERE AND NOT IN `propose.py`. That module's rules stay OFF
#: by default: they are what the golden fixtures and every published
#: reproduction block are pinned to, and the proposer's own CLI is the
#: measuring instrument. What moved is this COMMAND's default, and the two
#: layers disagreeing on purpose is why the disagreement is written down here
#: rather than discovered later.
#:
#: Each arm keeps an explicit opt-out: `--outline-run 0` (the number was
#: always its own switch) and `--no-unit-fold`. A default a caller cannot turn
#: off is not a default.
#: Arm E joined the default on 2026-09-08, one round after the other two, on a
#: measurement taken AFTER the first move rather than before it: Arm F's
#: published "5 of 12" was measured with Arm E on, and the round-3 default --
#: D plus F, Arm E off -- scored 2 of 12 with `docx` at 0 of 3. The fold's
#: table clause folds a table back into the heading that introduces it, and
#: with Arm E off a grid table is not one block but one block per rule line,
#: so there was nothing whole to fold. `--no-table-grid` is its opt-out.
DEFAULT_OUTLINE_RUN = 3
DEFAULT_TABLE_GRID = True
DEFAULT_UNIT_FOLD = True
#: Round 6's two, and they moved together on 2026-09-09 because neither carries
#: the reference on its own: G1+G2 takes `pdf` from 2 of 8 to 5 of 8, and
#: `--drop-wrapped-outline` -- measured on its own in round 3 as a 1-document
#: change and shipped off -- takes the pair to 7 of 8. Each keeps its explicit
#: opt-out (`--no-outline-gate`, `--keep-wrapped-outline`), and the two of them
#: off reproduce the 2026-09-08 default byte for byte. The acceptance the move
#: had to clear was not the reference alone: hit@8 holds rank 1 on every row it
#: held before, on every bundle measured. Both measurements are in
#: `docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md`.
DEFAULT_OUTLINE_GATE = True
DEFAULT_DROP_WRAPPED_OUTLINE = True
#: Round 6 built it and left it off for want of one number; round 7 measured
#: that number and moved it (2026-09-10). Measured over the 39-document corpus:
#: every one of the 32 documents that gets a plan lost the text above its first
#: concept -- 159 704 characters, 9.18 % of the corpus, 45 841 from a single
#: document. It adds NO boundary; it moves one span's start, so it is a repair
#: to coverage rather than a segmentation rule, which is why it sits apart from
#: the arms above.
#:
#: The acceptance it had to clear was the same one round 6 set: the 12-position
#: reference does not move (it changes not one cell, 7 of 8 `pdf`, 3 of 3
#: `docx`, 10 of 12) AND hit@8 holds rank 1 on every row it held, measured end
#: to end on a K2 bundle built with it. Opt-out `--no-first-span-from-zero`.
DEFAULT_FIRST_SPAN_FROM_ZERO = True
#: Round 8. The remaining coverage gap round 7 named and left open -- 43 631
#: characters, 2.51 % of the corpus over 8 of 32 documents with a plan -- has
#: ONE cause: three steps remove a candidate AFTER its neighbour's span was
#: already closed against it, so the removed mark's text is in no segment. The
#: rule closes each span against the next SURVIVOR. It adds no boundary, so the
#: concept count is unchanged and the cost to a consumer is a re-run.
DEFAULT_CLOSE_SPAN_GAPS = True
#: Round 20 (K3-20). A section whose body is its heading alone gets `parent:`
#: naming the nearest ancestor that holds text (`propose._link_shells`) and,
#: since K3-21, one `Enclosing section:` link line in its body. NOT a
#: segmentation rule: it adds no boundary and copies no text, and the concept
#: count is unchanged. OFF, re-measured in K3-21 once `okf consume` read the
#: key -- the reason it was off before. On the one standard with shells hit@1,
#: hit@8 and hit@50 stay 6/6 at both k with the known-positive at rank 1, but
#: the link's bundle-absolute path repeats the document's directory in 675
#: bodies, and the delivered set moved on 2 of 8 questions at the default k
#: and 3 of 8 at k 50, newly delivered shells matching the question ONLY
#: through that path on 1 of 3, 4 of 7 and 6 of 23. That is the saturation
#: K3-20 took out of the id signal, re-entering through the body. It moves the
#: bytes of a bundle holding a heading-only section and of nothing else: 0 of
#: 453 K2 bodies are heading-only, and the five-document folder builds
#: byte-identical with it.
DEFAULT_SHELL_PARENT = False
#: Round 9. Clause 1 asked whether a title ENDED in a number, which is a
#: question about the number rather than about the title: a drawing's dimension
#: chain, a P&ID's schematic labels, a door schedule and a borehole log's
#: coordinate column all end in integers and name nothing. Measured over the
#: 43-document corpus, clause 1 discarded 68 candidates over 11 of 39 readable
#: documents, and 19 of those over 5 of the 11 were data rows of that kind.
#: The rule requires a NAME to survive stripping the page number; the threshold
#: is swept rather than chosen (`propose.CONTENTS_NAME_RUN`). It rescues 16 of
#: the 19 and 0 of the 49 real contents entries -- the three it does not rescue
#: carry a real word (`- 8.225m Fjell 41`) and this predicate cannot separate
#: them from a contents entry. It only ever REMOVES members from a run, so it
#: can only add concepts, never take one away. Opt-out `--no-contents-name`.
DEFAULT_CONTENTS_NAME = True
#: Round 10's rule for the type that declares nothing. `rtf` came back at 0 of
#: 0 declared headings, 0 concepts and 1368 of 1368 characters in no segment:
#: the container has no heading style, so the author's title is bold text.
#: The grammar is markdown, not `rtf` -- the converter already writes that
#: title as `**...**` in the same output every office row produces -- and it is
#: gated by the principle Arm D already carries, that recovery yields to
#: declaration. Three parameters were swept over 47 readable documents and one
#: carried (see `tests/test_bold_title.py`). Default set by measurement below.
DEFAULT_BOLD_TITLE = False
#: Round 3's two spreadsheet rules (D1 and D3), held back through rounds 5 and
#: 6 by a RETRIEVAL regression rather than by the reference: they take the
#: sheet to 11 of 12 and `xlsx` to its fasit, and on a K2 bundle built with
#: them row 1 of the hit@8 set fell from rank 1 to rank 2.
#:
#: Round 7 decomposed that fall and it was never these rules' fault. Splitting
#: the gold document from 1 concept into 12 makes its own 12 concepts occupy
#: the document-prior signal's entire top tie group, so the concept leading the
#: body signal takes prior position 11 instead of 1 -- RRF reading a tie-break
#: by `concept_id` as if it were a measurement. `consume.DEFAULT_TIE_SHARED_RANK`
#: is the repair, and with it in place all three of round 7's acceptance
#: conditions hold at once. Opt-outs `--no-sheet-section-rows`,
#: `--no-keep-table-heading`.
DEFAULT_SHEET_SECTION_ROWS = True
DEFAULT_KEEP_TABLE_HEADING = True
#: Round 4's two PDF READERS, both off. They are not arms and they are not in
#: the paragraph above: an arm changes how the proposer cuts a text, and these
#: two change what the text SAYS -- so they sit before every arm rather than
#: beside them, and a measurement of an arm is a measurement of it reading
#: whatever these produced.
#:
#: `DEFAULT_PDF_HEADINGS` is off BY MEASUREMENT and the measurement is in
#: `docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md`, not by caution.
#: `DEFAULT_OCR` is off BY CONSTRUCTION and can never move: the engine is an
#: optional dependency group, so a default that needed it would make the
#: package's ordinary install fail on the first scanned page.
DEFAULT_PDF_HEADINGS = False
DEFAULT_OCR = False
#: Round 5's third value on the SAME axis, and off for a third reason: not
#: caution, not construction, but a reachable set with no reference in it.
#: The reserve reads typography only where the outline gate admits no run, so
#: over the 43-document corpus it can act on 4 of 39 readable documents -- and
#: on the twelve-position reference it changes NOTHING, because the only
#: positions where it fires are one PDF whose glyphs carry no ToUnicode
#: mapping and four office documents the PDF reader never touches. The
#: position it was built for numbers its own chapters, so the reserve is
#: silent there by construction. Measured in
#: `docs/2026-09-08-k3-runde5-hitat8-og-skriftakse.md`.
DEFAULT_PDF_HEADINGS_RESERVE = False
#: The PDF bookmark arm. OFF, and this round did not move it: the arm was
#: measured on ONE document, and a default that changes every PDF bundle a
#: consumer holds is an operator's call with the numbers in front of them.
#: It is BYTE-IDENTICAL on a PDF that carries no `/Outlines`, which is the
#: common case and the reason the flag is safe to hand out before that call.
DEFAULT_PDF_OUTLINE = False
#: The timestamp written when the caller passes none, for the ingest stamp and
#: the proposal stamp alike. ONE constant: two independently-defaulted literals
#: drift, and the drift shows up only as two bundles differing in a field
#: nobody set.
DEFAULT_STAMP = "1970-01-01T00:00:00Z"
def frontmatter_from_flags(pairs: Sequence[str]) -> dict[str, str]:
"""`--frontmatter KEY=VALUE`, split on the FIRST `=` and only there.
The first `=` because the value is the one that needs the rest: a
publisher's address carries `?languageCode=nb`, and a split on every `=`
would cut the `sources` flow mapping in half. Key and value are validated
by the door (`validate_concept_frontmatter`); this only refuses what is not
a pair, and a key named twice -- which of two values was meant is a guess.
"""
values: dict[str, str] = {}
for pair in pairs:
key, sep, value = pair.partition("=")
if not sep:
raise IngestError(
f"--frontmatter {pair!r} has no '='; the form is KEY=VALUE",
code="run_frontmatter_invalid",
)
if key in values:
raise IngestError(
f"--frontmatter names {key!r} twice; refusing to pick one of the two values",
code="run_frontmatter_invalid",
)
values[key] = value
return values
def _document_prefixes(inbox: Path, walked: Sequence[Path]) -> dict[Path, str]:
"""Each document's directory: the name it declares, else its file name.
MEASURED: a NISO-STS delivery landed every one of its 2 761 concepts under
a directory named for the delivery path's file name, a UUID occurring 0
times in the document, while the document's own `<doc-number>` said what it
was. Only the file's STEM is replaced; the folders above it are the
operator's arrangement and stay.
A declared name another document in this run also claims -- by declaring
it, or by its file name reducing to it -- is not taken by either, and both
keep their file name. The gate Door B already has would refuse both and
tell the operator to rename one, and a name read from inside a document is
not one a rename can change. Said on stderr rather than silently, because a
directory that stays a UUID is otherwise indistinguishable from this rule
never having run.
"""
def scope(prefix: str) -> str:
return "/".join(reduce_to_id_grammar(part) for part in prefix.split("/"))
named = {source: source.relative_to(inbox).with_suffix("").as_posix() for source in walked}
declared: dict[Path, tuple[str, str]] = {}
for source in walked:
try:
identity = declared_identity(source.name, source.read_bytes())
except OSError:
# The proposer reads the same file next and reports it per file.
continue
if identity is None or identity.doc_number is None:
continue
slug = reduce_to_id_grammar(identity.doc_number)
if slug:
parent = source.relative_to(inbox).parent
declared[source] = ((parent / slug).as_posix(), identity.doc_number)
claims: dict[str, set[Path]] = {}
for source in walked:
claims.setdefault(scope(named[source]), set()).add(source)
for source, (prefix, _) in declared.items():
claims.setdefault(scope(prefix), set()).add(source)
prefixes = dict(named)
for source, (prefix, doc_number) in declared.items():
others = sorted(named[other] for other in claims[scope(prefix)] if other != source)
if others:
print(
f"{CLI_ID}: {named[source]}: <doc-number> {doc_number!r} names {prefix!r}, "
f"which {', '.join(others)} also claims; both keep their file name",
file=sys.stderr,
)
continue
prefixes[source] = prefix
return prefixes
def _propose_plans(
inbox: Path,
bundle: Path,
plans_dir: Path,
*,
proposed_at: str,
okf_type: str,
outline_run: int = 0,
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = DEFAULT_KEEP_TABLE_HEADING,
sheet_section_rows: bool = DEFAULT_SHEET_SECTION_ROWS,
drop_wrapped_outline: bool = False,
outline_gate: bool = False,
first_span_from_zero: bool = False,
close_span_gaps: bool = False,
contents_name: bool = False,
bold_title: bool = False,
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
shell_parent: bool = DEFAULT_SHELL_PARENT,
) -> tuple[int, int, int]:
"""Propose a plan per dropped file. Returns (written, nothing, failed).
The walk is the DOOR's walk, imported rather than restated, so the set of
documents that get a plan is exactly the set that gets ingested.
Neither "nothing to propose" nor "cannot read" stops the loop, because
neither stops the door: the first lands the document as one flat concept
and the second is a coded rejection that K1b accounts for. The shell loop
behaved the same way -- `exit 1` for the eleven with no boundary, `exit 2`
for the four unreadable -- and a build that aborted on either would refuse
corpora the two-script path completes.
"""
walked, _ = walk_inbox(inbox, exclude=bundle)
prefixes = _document_prefixes(inbox, walked)
written = nothing = failed = 0
for position, source in enumerate(walked, start=1):
relative = source.relative_to(inbox)
try:
outcome = propose_run(
source,
plans_dir / f"{position:02d}.json",
okf_type=okf_type,
proposed_at=proposed_at,
path_prefix=prefixes[source],
outline_run=outline_run,
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
outline_gate=outline_gate,
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
contents_name=contents_name,
bold_title=bold_title,
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
pdf_outline=pdf_outline,
shell_parent=shell_parent,
)
except ProposerError as exc:
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
failed += 1
continue
if outcome == 0:
written += 1
else:
nothing += 1
return (written, nothing, failed)
def build(
inbox: Path,
bundle: Path,
*,
ingested_at: str = DEFAULT_STAMP,
proposed_at: str | None = None,
bundle_id: str | None = None,
okf_version: str | None = None,
segments: bool = True,
plans_dir: Path | None = None,
okf_type: str = "reference",
outline_run: int = DEFAULT_OUTLINE_RUN,
table_grid: bool = DEFAULT_TABLE_GRID,
unit_fold: bool = DEFAULT_UNIT_FOLD,
keep_table_heading: bool = DEFAULT_KEEP_TABLE_HEADING,
sheet_section_rows: bool = DEFAULT_SHEET_SECTION_ROWS,
drop_wrapped_outline: bool = DEFAULT_DROP_WRAPPED_OUTLINE,
outline_gate: bool = DEFAULT_OUTLINE_GATE,
first_span_from_zero: bool = DEFAULT_FIRST_SPAN_FROM_ZERO,
close_span_gaps: bool = DEFAULT_CLOSE_SPAN_GAPS,
contents_name: bool = DEFAULT_CONTENTS_NAME,
bold_title: bool = DEFAULT_BOLD_TITLE,
pdf_headings: bool = DEFAULT_PDF_HEADINGS,
pdf_headings_reserve: bool = DEFAULT_PDF_HEADINGS_RESERVE,
ocr: bool = DEFAULT_OCR,
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
shell_parent: bool = DEFAULT_SHELL_PARENT,
frontmatter: Mapping[str, str] | None = None,
) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing.
Keyword-only with defaults, so a caller who takes this as an API keeps a
source-compatible call when a flag is added.
`proposed_at` defaults to `ingested_at` rather than independently to
`DEFAULT_STAMP`: a caller naming one clock is naming "when this ran", not
asking for two different clocks. Left independently defaulted, a caller who
set only `--ingested-at` stamped the unsegmented concepts (which read the
call's `ingested_at` directly) and left every segmented one -- which reads
`segment.ingested_at`, the plan's `proposed_at` -- on `DEFAULT_STAMP`.
Measured on K2: 11 of 629 concepts. Two independently-set clocks are still
reachable by passing `--proposed-at` explicitly.
"""
if proposed_at is None:
proposed_at = ingested_at
# Refused HERE, before one proposal is written: a stated key that is wrong
# is wrong for every document, and the proposal pass is the long half.
concept_values = validate_concept_frontmatter(
frontmatter or {}, profile=SEGMENTED_OKF_V0_2 if segments else STRUCTURED_V1
)
# Bound to THIS run's outline minimum, once, so the proposer and the door
# cannot be handed two different thresholds for the same question.
reserve = (
partial(heading_reserve_applies, outline_run=outline_run)
if pdf_headings_reserve and not pdf_headings
else None
)
if not segments:
report = measure(
inbox,
bundle,
ingested_at=ingested_at,
profile=STRUCTURED_V1,
pdf_headings=pdf_headings,
heading_reserve=reserve,
ocr=ocr,
concept_frontmatter_values=concept_values,
)
_write_log(bundle, report, profile=STRUCTURED_V1)
return report
if bundle_id is None or okf_version is None:
missing = ", ".join(
flag
for flag, value in (("--bundle-id", bundle_id), ("--okf-version", okf_version))
if value is None
)
raise IngestError(
f"{missing} is required unless --segments off; a profile names a key and the "
"caller owns its value",
code="manifest_invalid",
)
with tempfile.TemporaryDirectory(prefix="okf-plans-") as scratch:
target = plans_dir if plans_dir is not None else Path(scratch)
target.mkdir(parents=True, exist_ok=True)
written, nothing, failed = _propose_plans(
inbox,
bundle,
target,
proposed_at=proposed_at,
okf_type=okf_type,
outline_run=outline_run,
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
outline_gate=outline_gate,
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
contents_name=contents_name,
bold_title=bold_title,
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
pdf_outline=pdf_outline,
shell_parent=shell_parent,
)
print(
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
f"{failed} unreadable",
file=sys.stderr,
)
plans = load_plans(target)
report = measure(
inbox,
bundle,
ingested_at=ingested_at,
plans=plans,
profile=SEGMENTED_OKF_V0_2,
root_frontmatter_values={"okf_version": okf_version, "bundle_id": bundle_id},
pdf_headings=pdf_headings,
heading_reserve=reserve,
ocr=ocr,
concept_frontmatter_values=concept_values,
)
_write_log(bundle, report, profile=SEGMENTED_OKF_V0_2)
return report
def _write_log(bundle: Path, report: CorpusReport, *, profile: BundleProfile) -> None:
"""The section 9 log, into the BUNDLE, and the root index link to it.
Lifted verbatim from the harness's own `main` rather than reimplemented:
the log carries `N`, which is the one fact about a run the bundle cannot
otherwise recover, and a build that wrote a bundle without it would ship an
artifact whose conservation identity is uncheckable.
"""
bundle.mkdir(parents=True, exist_ok=True)
(bundle / LOG_NAME).write_text(report.render_log(), encoding="utf-8", newline="")
def parse_args(argv: list[str] | None) -> argparse.Namespace:
parser = argparse.ArgumentParser(
prog="okf",
description=(
"OKF bundle tooling. One folder in, one bundle out -- and one command "
"from there to a bundle you can ask a question of."
),
)
subcommands = parser.add_subparsers(dest="command", required=True)
# Registered for `okf --help` and for the unknown-command error. Their
# arguments are NOT declared here: `main` hands the rest of `argv` to the
# implementing module, which owns them. `add_help=False` keeps
# `okf consume --help` reaching that module's help rather than this stub's.
for delegated, blurb in (
("consume", "cut a bundle to one contract-conformant payload for one question"),
("check", "check a consumption skill and one payload against the contract"),
("skill", "instantiate the consumption skill template for one bundle"),
("project", "folder in, bundle plus skill out: build and skill in one step"),
):
subcommands.add_parser(delegated, help=blurb, add_help=False)
build_parser = subcommands.add_parser(
"build",
help="build an OKF bundle from a folder of documents",
description=(
"Walk a folder recursively, propose a segmentation for each document, "
"replay those proposals through Door B, and write the bundle. Reports "
"the conservation identity `merged + coded rejections == N` on stdout "
"and exits non-zero when it does not hold."
),
)
build_parser.add_argument("inbox", type=Path, help="the folder of documents to ingest")
build_parser.add_argument(
"--bundle", type=Path, required=True, help="where to write the OKF bundle"
)
build_parser.add_argument(
"--bundle-id",
default=None,
help="required unless --segments off: what a consumer joins the concepts on",
)
build_parser.add_argument(
"--okf-version",
default=None,
help=(
"required unless --segments off: the upstream OKF version this bundle "
"declares. An argument and never a constant -- the VALUE belongs to the "
"catalog (decision E1)"
),
)
build_parser.add_argument(
"--ingested-at",
default=DEFAULT_STAMP,
help=f"stamped verbatim. Default {DEFAULT_STAMP}: deterministic, never the clock",
)
build_parser.add_argument(
"--proposed-at",
default=None,
help=f"written into every proposal. Defaults to --ingested-at (itself {DEFAULT_STAMP} "
"if that is also omitted), so naming one clock never leaves the other unset",
)
build_parser.add_argument(
"--segments",
choices=("on", "off"),
default="on",
help=(
"on (the default) proposes a segmentation per document with the "
"mechanical rules and replays it; off ingests each document as one "
"concept and asks for no root values"
),
)
build_parser.add_argument(
"--okf-type", default="reference", help="okf_type for every concept and proposal"
)
build_parser.add_argument(
"--frontmatter",
action="append",
default=None,
metavar="KEY=VALUE",
help=(
"stamp KEY: VALUE on every concept of this run; repeatable. Split on "
"the FIRST '=' and written verbatim on ONE line, so a flow mapping such "
"as 'sources=[{ resource: <url>, title: <t> }]' survives whole. Adds "
"any key, and REPLACES only sources and description -- the two with "
"a layer below them (the document's own identity, else the file "
"name). Every other key the door writes itself is refused"
),
)
build_parser.add_argument(
"--plans-dir",
type=Path,
default=None,
help=(
"keep the proposals here instead of discarding them. Every entry is "
"PROPOSED, never adjudicated -- this is where an operator reads what "
"the run replayed"
),
)
build_parser.add_argument(
"--outline-run",
type=int,
default=DEFAULT_OUTLINE_RUN,
metavar="N",
help=(
"Arm D, passed to the proposer unchanged: also propose a boundary at "
"each line of the document's own numbered outline, where the integers "
"sustain an ascending run of at least N. The default is 3 (operator, "
"2026-09-08); pass 0 to turn the arm off, which is this arm's "
"explicit opt-out. Measured on a tender PDF whose headings are bare "
"integers: no boundary at 0, nine at 3"
),
)
build_parser.add_argument(
"--table-grid",
action="store_true",
default=DEFAULT_TABLE_GRID,
help=(
"Arm E, passed to the proposer unchanged: a pandoc grid-table rule "
"line no longer closes an open table block, so one grid table is one "
"concept instead of one per row group. ON by default since "
"2026-09-08 (operator); the flag is kept so a call site that passes "
"it stays valid. Measured on the K3 sample: it changes a `.docx` "
"experience list from 21 concepts to 6, and without it Arm F has no "
"joined table to fold"
),
)
build_parser.add_argument(
"--no-table-grid",
action="store_false",
dest="table_grid",
help=(
"Arm E's explicit opt-out. With --outline-run 0 and --no-unit-fold "
"it reproduces the pre-2026-09-08 default byte for byte"
),
)
build_parser.add_argument(
"--unit-fold",
action="store_true",
default=DEFAULT_UNIT_FOLD,
help=(
"Arm F, passed to the proposer unchanged: discard a contents-list run, "
"fold a deeper heading into its parent, fold a table back into the "
"shorter heading that introduces it. It adds no boundary, so it can "
"only reduce a plan. ON by default since 2026-09-08 (operator); the "
"flag is kept so a call site that passes it stays valid. Measured on "
"the K3 sample: 5 of 12 documents match the operator's unit "
"worksheet, against 2 for the arm below it, with no cell worse"
),
)
build_parser.add_argument(
"--no-unit-fold",
action="store_false",
dest="unit_fold",
help=(
"Arm F's explicit opt-out. With --outline-run 0 it reproduces the "
"pre-2026-09-08 default byte for byte, which is what keeps the move "
"reversible for a consumer who needs the old bytes"
),
)
build_parser.add_argument(
"--keep-table-heading",
action="store_true",
default=DEFAULT_KEEP_TABLE_HEADING,
help=(
"D1, passed to the proposer unchanged: keep a heading whose body is "
"empty only because a table opens under it, and absorb that table "
"into its span. ON since 2026-09-10. Measured on a tender price "
"sheet: the concept count does not move (1 -> 1) and the concept "
"gains the heading line it was missing"
),
)
build_parser.add_argument(
"--no-keep-table-heading",
action="store_false",
dest="keep_table_heading",
help="D1's explicit opt-out. A default a caller cannot turn off is not a default",
)
build_parser.add_argument(
"--sheet-section-rows",
action="store_true",
default=DEFAULT_SHEET_SECTION_ROWS,
help=(
"D3, passed to the proposer unchanged: cut an open table block at "
"the rows that label its sections -- a RUN of rows whose first cell "
"is a bare numeric label. The opposite direction from Arm E, which "
"decides how far a block extends; this decides where it is cut "
"inside. ON since 2026-09-10, once the fusion stopped punishing the "
"split it produces. Measured on a tender price sheet whose whole "
"body is one table block: 1 concept -> 12, against a reference of "
"11 cost groups plus the sheet's preamble"
),
)
build_parser.add_argument(
"--no-sheet-section-rows",
action="store_false",
dest="sheet_section_rows",
help="D3's explicit opt-out. With --no-keep-table-heading it reproduces the pre-2026-09-10 bytes",
)
build_parser.add_argument(
"--drop-wrapped-outline",
action="store_true",
default=DEFAULT_DROP_WRAPPED_OUTLINE,
help=(
"D3, passed to the proposer unchanged: do not admit an outline "
"candidate whose line continues onto the next one. Recovered "
"candidates only, never a heading the document declares. Absent "
"(the default) is OFF. Measured on the K3 sample: 8 of 34 outline "
"candidates wrap, and they are the quoted regulation paragraphs "
"and the risk-table rows the operator wrote off"
),
)
build_parser.add_argument(
"--keep-wrapped-outline",
action="store_false",
dest="drop_wrapped_outline",
help=(
"The opt-out for the rule above. With --no-outline-gate it "
"reproduces the pre-2026-09-09 default byte for byte"
),
)
build_parser.add_argument(
"--outline-gate",
action="store_true",
default=DEFAULT_OUTLINE_GATE,
help=(
"G1+G2, passed to the proposer unchanged: admit Arm D's RECOVERED "
"headings only where the document declares none of its own, plus "
"any one recovered heading whose span covers OUTLINE_SHARE of the "
"text. It only removes marks, and it removes them before spans are "
"closed, so no character falls outside every segment. Measured on "
"the K3 sample: `pdf` 2 of 8 -> 5 of 8 alone, 7 of 8 with "
"--drop-wrapped-outline, `docx` unchanged at 3 of 3"
),
)
build_parser.add_argument(
"--no-outline-gate",
action="store_false",
dest="outline_gate",
help="The gate's explicit opt-out. A default a caller cannot turn off is not a default",
)
build_parser.add_argument(
"--first-span-from-zero",
action="store_true",
default=DEFAULT_FIRST_SPAN_FROM_ZERO,
help=(
"Start the first concept at character 0, so the text above it is "
"carried by a segment instead of by none. Measured over the "
"39-document corpus: 32 of the 32 documents with a plan lost "
"their opening before this moved -- 159 704 characters, 9.18 "
"%% of the corpus. It adds no boundary and moves one span's start. "
"ON since 2026-09-10"
),
)
build_parser.add_argument(
"--no-first-span-from-zero",
action="store_false",
dest="first_span_from_zero",
help=(
"The repair's explicit opt-out, for a consumer who needs the "
"pre-2026-09-10 first span byte for byte"
),
)
build_parser.add_argument(
"--close-span-gaps",
action="store_true",
default=DEFAULT_CLOSE_SPAN_GAPS,
help=(
"Close a concept's span against the next SURVIVING concept, and "
"the last one against the end of the text, so a candidate removed "
"after its neighbour's span was fixed does not take its text out "
"of the plan. Measured over the 39-document corpus: it closes the "
"43 631 characters -- 2.51 %% of the corpus over 8 of the 32 "
"documents with a plan -- that round 7 left, both the 26 041 "
"between entries and the 17 590 after the last. It adds no "
"boundary and moves only spans' ends. ON since 2026-09-11"
),
)
build_parser.add_argument(
"--no-close-span-gaps",
action="store_false",
dest="close_span_gaps",
help=(
"The repair's explicit opt-out, for a consumer who needs the "
"pre-2026-09-11 spans byte for byte"
),
)
build_parser.add_argument(
"--shell-parent",
action="store_true",
default=DEFAULT_SHELL_PARENT,
help=(
"Give a concept whose body is its heading alone a `parent:` naming "
"the nearest ancestor that holds text, by the plan's level and "
"order, passing over an empty ancestor. Nothing is copied and no "
"boundary moves. Measured on one standard: 710 of 2 761 concepts "
"are heading-only, and the route names the ancestor its own "
"nesting names on 708 of them. OFF: `okf consume` does not read "
"the key"
),
)
build_parser.add_argument(
"--no-shell-parent",
action="store_false",
dest="shell_parent",
help="The explicit form of the default",
)
build_parser.add_argument(
"--contents-name",
action="store_true",
default=DEFAULT_CONTENTS_NAME,
help=(
"Admit a title into a contents run only when a NAME survives "
"stripping its trailing page number, so a run of data rows is not "
"read as a contents listing and discarded. Measured over the "
"39-document corpus: clause 1 dropped 68 candidates over 11 "
"documents, of which 19 over 5 documents were data rows; the rule "
"rescues 16 of the 19 and 0 of the 49 real contents entries. It "
"only removes members from a run, so it can only add concepts. "
"ON since 2026-09-09"
),
)
build_parser.add_argument(
"--no-contents-name",
action="store_false",
dest="contents_name",
help=(
"The repair's explicit opt-out, for a consumer who needs the "
"pre-2026-09-09 contents-run predicate byte for byte"
),
)
build_parser.add_argument(
"--bold-title",
action="store_true",
default=DEFAULT_BOLD_TITLE,
help=(
"Read a line that is ONE bold span as a title, in a document that "
"declares no heading of its own. For the type whose container has "
"no heading style at all: `rtf` reached round 9 at 0 of 0 declared "
"headings, 0 concepts and 1368 of 1368 characters in no segment. "
"The grammar is markdown, so it reaches every type the converter "
"writes bold for, and it is inert for `pdf`, which never goes "
"through the converter. Measured over 47 readable documents: 0 "
"false positives on the 31 that declare, by the gate"
),
)
build_parser.add_argument(
"--no-bold-title",
action="store_false",
dest="bold_title",
help="The rule's explicit opt-out",
)
build_parser.add_argument(
"--pdf-headings",
choices=("none", "font", "font-reserve"),
default=(
"font"
if DEFAULT_PDF_HEADINGS
else "font-reserve"
if DEFAULT_PDF_HEADINGS_RESERVE
else "none"
),
help=(
"how a PDF's headings are recovered before any arm reads the text. "
"none (the default) is today's extraction: a PDF carries no heading "
"markup and none is inferred. font infers one from TYPOGRAPHY -- a "
"line whose dominant size is above the document's body median AND "
"whose dominant font name says bold -- and emits it as an ATX "
"heading in the same markdown the office path produces, so the "
"existing heading rule reads it and no PDF-only grammar exists. A "
"CONJUNCTION, measured: adding weight as a disjunct took precision "
"from 0.786 to 0.524. font-reserve is font applied ONLY to a "
"document whose own numbering the outline arm finds nothing of -- "
"typography as a second source where there is no first one, never "
"on top of one. Three values of one option, so no caller can ask "
"for two at once"
),
)
build_parser.add_argument(
"--pdf-outline",
action="store_true",
default=DEFAULT_PDF_OUTLINE,
help=(
"OFF. Cut a PDF at the boundaries its own /Outlines bookmark tree "
"declares, instead of at the ones the text rules recover. It is a "
"SEGMENTATION arm and not a reader option: the extracted text is "
"byte for byte the same either way, and a PDF that carries no "
"bookmark tree builds byte-identically with the flag on. Measured "
"on one 701-page process code whose publisher also ships a NISO-STS "
"structure for it: the text rules recover 1967 of 2761 titled "
"sections and 0 of its 28 chapters, while its bookmark tree matches "
"2761 of 2761 exactly. The title comes from the BOOKMARK, so it is "
"not cut short at the page's line break, and a page before the "
"first bookmark destination is the table of contents rather than a "
"second copy of the body. ONE document, ONE format, ONE publisher, "
"and a bookmark tree is the publisher's CLAIM about its own "
"structure -- a stale or wrong tree carries its error straight into "
"the segmentation"
),
)
build_parser.add_argument(
"--no-pdf-outline",
action="store_false",
dest="pdf_outline",
help="The arm's explicit opt-out",
)
build_parser.add_argument(
"--ocr",
action="store_true",
default=DEFAULT_OCR,
help=(
"read a PDF page as an IMAGE when its own text never arrived -- the "
"page extracts empty, or as a page of (cid:N) placeholder codes. "
"Requires the optional 'ocr' group (pip install "
"'llm-ingestion-okf[extract,ocr]'); without it every affected file "
"is a coded rejection, never a crash. Never a default and never "
"able to become one: the engine is an optional dependency. OCR text "
"is a reading of a picture, so it is deterministic within one model "
"version and not across one"
),
)
build_parser.add_argument("--report", type=Path, default=None, help="also write the report")
return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> int:
arguments = list(sys.argv[1:] if argv is None else argv)
if arguments and arguments[0] in DELEGATED:
return _delegate(arguments[0], arguments[1:])
args = parse_args(arguments)
if not args.inbox.is_dir():
print(f"{CLI_ID}: FAILED - no such folder: {args.inbox}", file=sys.stderr)
return 2
try:
report = build(
args.inbox,
args.bundle,
ingested_at=args.ingested_at,
proposed_at=args.proposed_at,
bundle_id=args.bundle_id,
okf_version=args.okf_version,
segments=args.segments == "on",
plans_dir=args.plans_dir,
okf_type=args.okf_type,
outline_run=args.outline_run,
table_grid=args.table_grid,
unit_fold=args.unit_fold,
keep_table_heading=args.keep_table_heading,
sheet_section_rows=args.sheet_section_rows,
drop_wrapped_outline=args.drop_wrapped_outline,
outline_gate=args.outline_gate,
first_span_from_zero=args.first_span_from_zero,
close_span_gaps=args.close_span_gaps,
contents_name=args.contents_name,
bold_title=args.bold_title,
pdf_headings=args.pdf_headings == "font",
pdf_headings_reserve=args.pdf_headings == "font-reserve",
ocr=args.ocr,
pdf_outline=args.pdf_outline,
shell_parent=args.shell_parent,
frontmatter=frontmatter_from_flags(args.frontmatter or ()),
)
except (IngestError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)
return 2
if args.report is not None:
args.report.parent.mkdir(parents=True, exist_ok=True)
args.report.write_text(report.render(), encoding="utf-8", newline="")
print(report.render())
if report.unaccounted or report.merged + report.rejected != report.n:
print(
f"{CLI_ID}: K1b FAILED - merged ({report.merged}) + coded rejections "
f"({report.rejected}) != N ({report.n}). Unaccounted: "
f"{', '.join(report.unaccounted) or '(none named)'}",
file=sys.stderr,
)
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
Reference in a new issue View git blame Copy permalink