"""`okf` — the installed command. Five subcommands: the whole chain, packaged. `build` a bundle from a folder, `consume` it for one question, `check` the payload against the contract, generate a `skill` for it, or do the whole thing in one step with `project`. Four of the five moved here on 2026-09-08 (O5) from unpackaged scripts under `tools/`, which a consumer who installs this library does not have; until then the reading direction existed only for someone standing in a clone. ## What it replaces Until this module existed, "run the door over a folder" was a shell loop over `tools/okf_propose_segments.py` followed by `tools/okf_corpus_run.py`, with nine flags between them and a `--path-prefix` rule that lived in a code block in a measurement report. Neither script was packaged, so the whole path was reachable only from a clone -- a consumer who installed this library could call `process_inbox` but could not run the thing the reports measured. `okf build --bundle ` is that path, packaged. It is ORCHESTRATION and nothing else: every rule it applies belongs to `propose` or `corpus`, which are the same modules `tools/` now calls. There is one implementation of each, and it is the packaged one. ## The two decisions this layer owns **The scope of a document's proposed paths is its RELATIVE PATH, minus the extension.** Section numbering is document-local, so two documents propose the same path and Door B refuses both; the loop in the reports passed the BASENAME, which was right while the inbox was flat. It stopped being right when the door started walking recursively: `a/krav.pdf` and `b/krav.pdf` would both reduce to `krav` and collide, which is the collision Door B is supposed to make impossible rather than merely detectable. For a flat inbox the relative path IS the basename, which is why the published bundles' bytes do not move. **Omitted timestamps do not come from the clock.** `--ingested-at` defaults to `DEFAULT_STAMP`, and `--proposed-at` defaults to `--ingested-at` -- so a caller who names one clock stamps the whole bundle with it, and a caller who names neither gets `DEFAULT_STAMP` for both, one constant either way. A wall-clock default would put a changing byte into the artifact and break rebuild-equals-incremental (K6) for every caller who did not pass the flags -- the property the segmented bundle is built on, and the one a convenience default is most likely to take away silently. The epoch is deliberate and readable as what it is: a stamp nobody set. A caller who wants the proposal and the replay dated differently passes both explicitly. ## What it does not decide **Which arm ships as the default is the operator's decision and is not taken here.** Three of the six are ON as of 2026-09-08 -- `--outline-run 3`, `--table-grid` and `--unit-fold` -- each moved by the operator on a measurement and each with an explicit opt-out. The remaining three are OFF unless the caller asks. This module's job is to state the answer, never to pick it. The arms are, however, REACHABLE from here, and that is a change of 2026-09-08. Until then `_propose_plans` called the proposer with no arm flag at all, so `okf build` ran Arm B while `tools/okf_propose_segments.py` could run Arm D, E and F -- a build path a full arm behind the proposer, reachable only by retyping the loop the packaging removed. Measured on a five-document folder: one tender PDF lands as ONE concept from the build path and as NINE with `--outline-run 3`, and nine is what the operator's unit worksheet asks for. Arm C (`--max-segment-chars`) stays unexposed: it is a character cap whose value nobody has measured against a reference, so it has no number to offer a caller. """ from __future__ import annotations import argparse import sys import tempfile from functools import partial from pathlib import Path from .corpus import LOG_NAME, CorpusReport, load_plans, measure from .errors import IngestError from .inbox import walk_inbox from .profiles import SEGMENTED_OKF_V0_2, STRUCTURED_V1, BundleProfile from .propose import ProposerError, heading_reserve_applies from .propose import run as propose_run __all__ = ["DEFAULT_STAMP", "build", "main", "measure"] #: The subcommands whose FLAGS belong to the module that implements them. #: #: Dispatched before `argparse` runs rather than rebuilt as subparsers here: a #: second declaration of `--reserve-top-rank` or `--rarity-weight` is a second #: place they can drift, and the pre-pass's flag set is the thing four #: measurement reports are pinned to. Each entry is registered as a subparser #: below anyway, so `okf --help` lists it and an unknown command is still an #: error -- the parser knows the NAMES, the modules keep the flags. #: #: Imported lazily inside the dispatch: `okf build` should not pay to import #: the ranker, and `okf consume` should not pay to import the proposer. DELEGATED = ("consume", "check", "skill", "project") def _delegate(command: str, argv: list[str]) -> int: if command == "consume": from .consume import main as run elif command == "check": from .contract_check import main as run elif command == "skill": from .skill import main as run else: from .project import main as run return run(argv) CLI_ID = "okf build" #: What `okf build` runs when no flag is given. Moved 2026-09-08 by the #: operator, from Arm B to Arm D plus Arm F, on the round-2 and round-3 #: measurements: the previous default was a full arm behind the proposer on #: the one document with a reference, and the fold matches 5 of 12 documents #: against the default's 2 with no cell worse. #: #: THE VALUE LIVES HERE AND NOT IN `propose.py`. That module's rules stay OFF #: by default: they are what the golden fixtures and every published #: reproduction block are pinned to, and the proposer's own CLI is the #: measuring instrument. What moved is this COMMAND's default, and the two #: layers disagreeing on purpose is why the disagreement is written down here #: rather than discovered later. #: #: Each arm keeps an explicit opt-out: `--outline-run 0` (the number was #: always its own switch) and `--no-unit-fold`. A default a caller cannot turn #: off is not a default. #: Arm E joined the default on 2026-09-08, one round after the other two, on a #: measurement taken AFTER the first move rather than before it: Arm F's #: published "5 of 12" was measured with Arm E on, and the round-3 default -- #: D plus F, Arm E off -- scored 2 of 12 with `docx` at 0 of 3. The fold's #: table clause folds a table back into the heading that introduces it, and #: with Arm E off a grid table is not one block but one block per rule line, #: so there was nothing whole to fold. `--no-table-grid` is its opt-out. DEFAULT_OUTLINE_RUN = 3 DEFAULT_TABLE_GRID = True DEFAULT_UNIT_FOLD = True #: Round 6's two, and they moved together on 2026-09-09 because neither carries #: the reference on its own: G1+G2 takes `pdf` from 2 of 8 to 5 of 8, and #: `--drop-wrapped-outline` -- measured on its own in round 3 as a 1-document #: change and shipped off -- takes the pair to 7 of 8. Each keeps its explicit #: opt-out (`--no-outline-gate`, `--keep-wrapped-outline`), and the two of them #: off reproduce the 2026-09-08 default byte for byte. The acceptance the move #: had to clear was not the reference alone: hit@8 holds rank 1 on every row it #: held before, on every bundle measured. Both measurements are in #: `docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md`. DEFAULT_OUTLINE_GATE = True DEFAULT_DROP_WRAPPED_OUTLINE = True #: Round 6 built it and left it off for want of one number; round 7 measured #: that number and moved it (2026-09-10). Measured over the 39-document corpus: #: every one of the 32 documents that gets a plan lost the text above its first #: concept -- 159 704 characters, 9.18 % of the corpus, 45 841 from a single #: document. It adds NO boundary; it moves one span's start, so it is a repair #: to coverage rather than a segmentation rule, which is why it sits apart from #: the arms above. #: #: The acceptance it had to clear was the same one round 6 set: the 12-position #: reference does not move (it changes not one cell, 7 of 8 `pdf`, 3 of 3 #: `docx`, 10 of 12) AND hit@8 holds rank 1 on every row it held, measured end #: to end on a K2 bundle built with it. Opt-out `--no-first-span-from-zero`. DEFAULT_FIRST_SPAN_FROM_ZERO = True #: Round 8. The remaining coverage gap round 7 named and left open -- 43 631 #: characters, 2.51 % of the corpus over 8 of 32 documents with a plan -- has #: ONE cause: three steps remove a candidate AFTER its neighbour's span was #: already closed against it, so the removed mark's text is in no segment. The #: rule closes each span against the next SURVIVOR. It adds no boundary, so the #: concept count is unchanged and the cost to a consumer is a re-run. DEFAULT_CLOSE_SPAN_GAPS = True #: Round 9. Clause 1 asked whether a title ENDED in a number, which is a #: question about the number rather than about the title: a drawing's dimension #: chain, a P&ID's schematic labels, a door schedule and a borehole log's #: coordinate column all end in integers and name nothing. Measured over the #: 43-document corpus, clause 1 discarded 68 candidates over 11 of 39 readable #: documents, and 19 of those over 5 of the 11 were data rows of that kind. #: The rule requires a NAME to survive stripping the page number; the threshold #: is swept rather than chosen (`propose.CONTENTS_NAME_RUN`). It rescues 16 of #: the 19 and 0 of the 49 real contents entries -- the three it does not rescue #: carry a real word (`- 8.225m Fjell 41`) and this predicate cannot separate #: them from a contents entry. It only ever REMOVES members from a run, so it #: can only add concepts, never take one away. Opt-out `--no-contents-name`. DEFAULT_CONTENTS_NAME = True #: Round 10's rule for the type that declares nothing. `rtf` came back at 0 of #: 0 declared headings, 0 concepts and 1368 of 1368 characters in no segment: #: the container has no heading style, so the author's title is bold text. #: The grammar is markdown, not `rtf` -- the converter already writes that #: title as `**...**` in the same output every office row produces -- and it is #: gated by the principle Arm D already carries, that recovery yields to #: declaration. Three parameters were swept over 47 readable documents and one #: carried (see `tests/test_bold_title.py`). Default set by measurement below. DEFAULT_BOLD_TITLE = False #: Round 3's two spreadsheet rules (D1 and D3), held back through rounds 5 and #: 6 by a RETRIEVAL regression rather than by the reference: they take the #: sheet to 11 of 12 and `xlsx` to its fasit, and on a K2 bundle built with #: them row 1 of the hit@8 set fell from rank 1 to rank 2. #: #: Round 7 decomposed that fall and it was never these rules' fault. Splitting #: the gold document from 1 concept into 12 makes its own 12 concepts occupy #: the document-prior signal's entire top tie group, so the concept leading the #: body signal takes prior position 11 instead of 1 -- RRF reading a tie-break #: by `concept_id` as if it were a measurement. `consume.DEFAULT_TIE_SHARED_RANK` #: is the repair, and with it in place all three of round 7's acceptance #: conditions hold at once. Opt-outs `--no-sheet-section-rows`, #: `--no-keep-table-heading`. DEFAULT_SHEET_SECTION_ROWS = True DEFAULT_KEEP_TABLE_HEADING = True #: Round 4's two PDF READERS, both off. They are not arms and they are not in #: the paragraph above: an arm changes how the proposer cuts a text, and these #: two change what the text SAYS -- so they sit before every arm rather than #: beside them, and a measurement of an arm is a measurement of it reading #: whatever these produced. #: #: `DEFAULT_PDF_HEADINGS` is off BY MEASUREMENT and the measurement is in #: `docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md`, not by caution. #: `DEFAULT_OCR` is off BY CONSTRUCTION and can never move: the engine is an #: optional dependency group, so a default that needed it would make the #: package's ordinary install fail on the first scanned page. DEFAULT_PDF_HEADINGS = False DEFAULT_OCR = False #: Round 5's third value on the SAME axis, and off for a third reason: not #: caution, not construction, but a reachable set with no reference in it. #: The reserve reads typography only where the outline gate admits no run, so #: over the 43-document corpus it can act on 4 of 39 readable documents -- and #: on the twelve-position reference it changes NOTHING, because the only #: positions where it fires are one PDF whose glyphs carry no ToUnicode #: mapping and four office documents the PDF reader never touches. The #: position it was built for numbers its own chapters, so the reserve is #: silent there by construction. Measured in #: `docs/2026-09-08-k3-runde5-hitat8-og-skriftakse.md`. DEFAULT_PDF_HEADINGS_RESERVE = False #: The PDF bookmark arm. OFF, and this round did not move it: the arm was #: measured on ONE document, and a default that changes every PDF bundle a #: consumer holds is an operator's call with the numbers in front of them. #: It is BYTE-IDENTICAL on a PDF that carries no `/Outlines`, which is the #: common case and the reason the flag is safe to hand out before that call. DEFAULT_PDF_OUTLINE = False #: The timestamp written when the caller passes none, for the ingest stamp and #: the proposal stamp alike. ONE constant: two independently-defaulted literals #: drift, and the drift shows up only as two bundles differing in a field #: nobody set. DEFAULT_STAMP = "1970-01-01T00:00:00Z" def _propose_plans( inbox: Path, bundle: Path, plans_dir: Path, *, proposed_at: str, okf_type: str, outline_run: int = 0, table_grid: bool = False, unit_fold: bool = False, keep_table_heading: bool = DEFAULT_KEEP_TABLE_HEADING, sheet_section_rows: bool = DEFAULT_SHEET_SECTION_ROWS, drop_wrapped_outline: bool = False, outline_gate: bool = False, first_span_from_zero: bool = False, close_span_gaps: bool = False, contents_name: bool = False, bold_title: bool = False, pdf_headings: bool = False, pdf_headings_reserve: bool = False, ocr: bool = False, pdf_outline: bool = DEFAULT_PDF_OUTLINE, ) -> tuple[int, int, int]: """Propose a plan per dropped file. Returns (written, nothing, failed). The walk is the DOOR's walk, imported rather than restated, so the set of documents that get a plan is exactly the set that gets ingested. Neither "nothing to propose" nor "cannot read" stops the loop, because neither stops the door: the first lands the document as one flat concept and the second is a coded rejection that K1b accounts for. The shell loop behaved the same way -- `exit 1` for the eleven with no boundary, `exit 2` for the four unreadable -- and a build that aborted on either would refuse corpora the two-script path completes. """ walked, _ = walk_inbox(inbox, exclude=bundle) written = nothing = failed = 0 for position, source in enumerate(walked, start=1): relative = source.relative_to(inbox) try: outcome = propose_run( source, plans_dir / f"{position:02d}.json", okf_type=okf_type, proposed_at=proposed_at, path_prefix=relative.with_suffix("").as_posix(), outline_run=outline_run, table_grid=table_grid, unit_fold=unit_fold, keep_table_heading=keep_table_heading, sheet_section_rows=sheet_section_rows, drop_wrapped_outline=drop_wrapped_outline, outline_gate=outline_gate, first_span_from_zero=first_span_from_zero, close_span_gaps=close_span_gaps, contents_name=contents_name, bold_title=bold_title, pdf_headings=pdf_headings, pdf_headings_reserve=pdf_headings_reserve, ocr=ocr, pdf_outline=pdf_outline, ) except ProposerError as exc: print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr) failed += 1 continue if outcome == 0: written += 1 else: nothing += 1 return (written, nothing, failed) def build( inbox: Path, bundle: Path, *, ingested_at: str = DEFAULT_STAMP, proposed_at: str | None = None, bundle_id: str | None = None, okf_version: str | None = None, segments: bool = True, plans_dir: Path | None = None, okf_type: str = "reference", outline_run: int = DEFAULT_OUTLINE_RUN, table_grid: bool = DEFAULT_TABLE_GRID, unit_fold: bool = DEFAULT_UNIT_FOLD, keep_table_heading: bool = DEFAULT_KEEP_TABLE_HEADING, sheet_section_rows: bool = DEFAULT_SHEET_SECTION_ROWS, drop_wrapped_outline: bool = DEFAULT_DROP_WRAPPED_OUTLINE, outline_gate: bool = DEFAULT_OUTLINE_GATE, first_span_from_zero: bool = DEFAULT_FIRST_SPAN_FROM_ZERO, close_span_gaps: bool = DEFAULT_CLOSE_SPAN_GAPS, contents_name: bool = DEFAULT_CONTENTS_NAME, bold_title: bool = DEFAULT_BOLD_TITLE, pdf_headings: bool = DEFAULT_PDF_HEADINGS, pdf_headings_reserve: bool = DEFAULT_PDF_HEADINGS_RESERVE, ocr: bool = DEFAULT_OCR, pdf_outline: bool = DEFAULT_PDF_OUTLINE, ) -> CorpusReport: """Folder in, bundle out. The whole command, minus argument parsing. Keyword-only with defaults, so a caller who takes this as an API keeps a source-compatible call when a flag is added. `proposed_at` defaults to `ingested_at` rather than independently to `DEFAULT_STAMP`: a caller naming one clock is naming "when this ran", not asking for two different clocks. Left independently defaulted, a caller who set only `--ingested-at` stamped the unsegmented concepts (which read the call's `ingested_at` directly) and left every segmented one -- which reads `segment.ingested_at`, the plan's `proposed_at` -- on `DEFAULT_STAMP`. Measured on K2: 11 of 629 concepts. Two independently-set clocks are still reachable by passing `--proposed-at` explicitly. """ if proposed_at is None: proposed_at = ingested_at # Bound to THIS run's outline minimum, once, so the proposer and the door # cannot be handed two different thresholds for the same question. reserve = ( partial(heading_reserve_applies, outline_run=outline_run) if pdf_headings_reserve and not pdf_headings else None ) if not segments: report = measure( inbox, bundle, ingested_at=ingested_at, profile=STRUCTURED_V1, pdf_headings=pdf_headings, heading_reserve=reserve, ocr=ocr, ) _write_log(bundle, report, profile=STRUCTURED_V1) return report if bundle_id is None or okf_version is None: missing = ", ".join( flag for flag, value in (("--bundle-id", bundle_id), ("--okf-version", okf_version)) if value is None ) raise IngestError( f"{missing} is required unless --segments off; a profile names a key and the " "caller owns its value", code="manifest_invalid", ) with tempfile.TemporaryDirectory(prefix="okf-plans-") as scratch: target = plans_dir if plans_dir is not None else Path(scratch) target.mkdir(parents=True, exist_ok=True) written, nothing, failed = _propose_plans( inbox, bundle, target, proposed_at=proposed_at, okf_type=okf_type, outline_run=outline_run, table_grid=table_grid, unit_fold=unit_fold, keep_table_heading=keep_table_heading, sheet_section_rows=sheet_section_rows, drop_wrapped_outline=drop_wrapped_outline, outline_gate=outline_gate, first_span_from_zero=first_span_from_zero, close_span_gaps=close_span_gaps, contents_name=contents_name, bold_title=bold_title, pdf_headings=pdf_headings, pdf_headings_reserve=pdf_headings_reserve, ocr=ocr, pdf_outline=pdf_outline, ) print( f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; " f"{failed} unreadable", file=sys.stderr, ) plans = load_plans(target) report = measure( inbox, bundle, ingested_at=ingested_at, plans=plans, profile=SEGMENTED_OKF_V0_2, root_frontmatter_values={"okf_version": okf_version, "bundle_id": bundle_id}, pdf_headings=pdf_headings, heading_reserve=reserve, ocr=ocr, ) _write_log(bundle, report, profile=SEGMENTED_OKF_V0_2) return report def _write_log(bundle: Path, report: CorpusReport, *, profile: BundleProfile) -> None: """The section 9 log, into the BUNDLE, and the root index link to it. Lifted verbatim from the harness's own `main` rather than reimplemented: the log carries `N`, which is the one fact about a run the bundle cannot otherwise recover, and a build that wrote a bundle without it would ship an artifact whose conservation identity is uncheckable. """ bundle.mkdir(parents=True, exist_ok=True) (bundle / LOG_NAME).write_text(report.render_log(), encoding="utf-8", newline="") def parse_args(argv: list[str] | None) -> argparse.Namespace: parser = argparse.ArgumentParser( prog="okf", description=( "OKF bundle tooling. One folder in, one bundle out -- and one command " "from there to a bundle you can ask a question of." ), ) subcommands = parser.add_subparsers(dest="command", required=True) # Registered for `okf --help` and for the unknown-command error. Their # arguments are NOT declared here: `main` hands the rest of `argv` to the # implementing module, which owns them. `add_help=False` keeps # `okf consume --help` reaching that module's help rather than this stub's. for delegated, blurb in ( ("consume", "cut a bundle to one contract-conformant payload for one question"), ("check", "check a consumption skill and one payload against the contract"), ("skill", "instantiate the consumption skill template for one bundle"), ("project", "folder in, bundle plus skill out: build and skill in one step"), ): subcommands.add_parser(delegated, help=blurb, add_help=False) build_parser = subcommands.add_parser( "build", help="build an OKF bundle from a folder of documents", description=( "Walk a folder recursively, propose a segmentation for each document, " "replay those proposals through Door B, and write the bundle. Reports " "the conservation identity `merged + coded rejections == N` on stdout " "and exits non-zero when it does not hold." ), ) build_parser.add_argument("inbox", type=Path, help="the folder of documents to ingest") build_parser.add_argument( "--bundle", type=Path, required=True, help="where to write the OKF bundle" ) build_parser.add_argument( "--bundle-id", default=None, help="required unless --segments off: what a consumer joins the concepts on", ) build_parser.add_argument( "--okf-version", default=None, help=( "required unless --segments off: the upstream OKF version this bundle " "declares. An argument and never a constant -- the VALUE belongs to the " "catalog (decision E1)" ), ) build_parser.add_argument( "--ingested-at", default=DEFAULT_STAMP, help=f"stamped verbatim. Default {DEFAULT_STAMP}: deterministic, never the clock", ) build_parser.add_argument( "--proposed-at", default=None, help=f"written into every proposal. Defaults to --ingested-at (itself {DEFAULT_STAMP} " "if that is also omitted), so naming one clock never leaves the other unset", ) build_parser.add_argument( "--segments", choices=("on", "off"), default="on", help=( "on (the default) proposes a segmentation per document with the " "mechanical rules and replays it; off ingests each document as one " "concept and asks for no root values" ), ) build_parser.add_argument( "--okf-type", default="reference", help="okf_type for every concept and proposal" ) build_parser.add_argument( "--plans-dir", type=Path, default=None, help=( "keep the proposals here instead of discarding them. Every entry is " "PROPOSED, never adjudicated -- this is where an operator reads what " "the run replayed" ), ) build_parser.add_argument( "--outline-run", type=int, default=DEFAULT_OUTLINE_RUN, metavar="N", help=( "Arm D, passed to the proposer unchanged: also propose a boundary at " "each line of the document's own numbered outline, where the integers " "sustain an ascending run of at least N. The default is 3 (operator, " "2026-09-08); pass 0 to turn the arm off, which is this arm's " "explicit opt-out. Measured on a tender PDF whose headings are bare " "integers: no boundary at 0, nine at 3" ), ) build_parser.add_argument( "--table-grid", action="store_true", default=DEFAULT_TABLE_GRID, help=( "Arm E, passed to the proposer unchanged: a pandoc grid-table rule " "line no longer closes an open table block, so one grid table is one " "concept instead of one per row group. ON by default since " "2026-09-08 (operator); the flag is kept so a call site that passes " "it stays valid. Measured on the K3 sample: it changes a `.docx` " "experience list from 21 concepts to 6, and without it Arm F has no " "joined table to fold" ), ) build_parser.add_argument( "--no-table-grid", action="store_false", dest="table_grid", help=( "Arm E's explicit opt-out. With --outline-run 0 and --no-unit-fold " "it reproduces the pre-2026-09-08 default byte for byte" ), ) build_parser.add_argument( "--unit-fold", action="store_true", default=DEFAULT_UNIT_FOLD, help=( "Arm F, passed to the proposer unchanged: discard a contents-list run, " "fold a deeper heading into its parent, fold a table back into the " "shorter heading that introduces it. It adds no boundary, so it can " "only reduce a plan. ON by default since 2026-09-08 (operator); the " "flag is kept so a call site that passes it stays valid. Measured on " "the K3 sample: 5 of 12 documents match the operator's unit " "worksheet, against 2 for the arm below it, with no cell worse" ), ) build_parser.add_argument( "--no-unit-fold", action="store_false", dest="unit_fold", help=( "Arm F's explicit opt-out. With --outline-run 0 it reproduces the " "pre-2026-09-08 default byte for byte, which is what keeps the move " "reversible for a consumer who needs the old bytes" ), ) build_parser.add_argument( "--keep-table-heading", action="store_true", default=DEFAULT_KEEP_TABLE_HEADING, help=( "D1, passed to the proposer unchanged: keep a heading whose body is " "empty only because a table opens under it, and absorb that table " "into its span. ON since 2026-09-10. Measured on a tender price " "sheet: the concept count does not move (1 -> 1) and the concept " "gains the heading line it was missing" ), ) build_parser.add_argument( "--no-keep-table-heading", action="store_false", dest="keep_table_heading", help="D1's explicit opt-out. A default a caller cannot turn off is not a default", ) build_parser.add_argument( "--sheet-section-rows", action="store_true", default=DEFAULT_SHEET_SECTION_ROWS, help=( "D3, passed to the proposer unchanged: cut an open table block at " "the rows that label its sections -- a RUN of rows whose first cell " "is a bare numeric label. The opposite direction from Arm E, which " "decides how far a block extends; this decides where it is cut " "inside. ON since 2026-09-10, once the fusion stopped punishing the " "split it produces. Measured on a tender price sheet whose whole " "body is one table block: 1 concept -> 12, against a reference of " "11 cost groups plus the sheet's preamble" ), ) build_parser.add_argument( "--no-sheet-section-rows", action="store_false", dest="sheet_section_rows", help="D3's explicit opt-out. With --no-keep-table-heading it reproduces the pre-2026-09-10 bytes", ) build_parser.add_argument( "--drop-wrapped-outline", action="store_true", default=DEFAULT_DROP_WRAPPED_OUTLINE, help=( "D3, passed to the proposer unchanged: do not admit an outline " "candidate whose line continues onto the next one. Recovered " "candidates only, never a heading the document declares. Absent " "(the default) is OFF. Measured on the K3 sample: 8 of 34 outline " "candidates wrap, and they are the quoted regulation paragraphs " "and the risk-table rows the operator wrote off" ), ) build_parser.add_argument( "--keep-wrapped-outline", action="store_false", dest="drop_wrapped_outline", help=( "The opt-out for the rule above. With --no-outline-gate it " "reproduces the pre-2026-09-09 default byte for byte" ), ) build_parser.add_argument( "--outline-gate", action="store_true", default=DEFAULT_OUTLINE_GATE, help=( "G1+G2, passed to the proposer unchanged: admit Arm D's RECOVERED " "headings only where the document declares none of its own, plus " "any one recovered heading whose span covers OUTLINE_SHARE of the " "text. It only removes marks, and it removes them before spans are " "closed, so no character falls outside every segment. Measured on " "the K3 sample: `pdf` 2 of 8 -> 5 of 8 alone, 7 of 8 with " "--drop-wrapped-outline, `docx` unchanged at 3 of 3" ), ) build_parser.add_argument( "--no-outline-gate", action="store_false", dest="outline_gate", help="The gate's explicit opt-out. A default a caller cannot turn off is not a default", ) build_parser.add_argument( "--first-span-from-zero", action="store_true", default=DEFAULT_FIRST_SPAN_FROM_ZERO, help=( "Start the first concept at character 0, so the text above it is " "carried by a segment instead of by none. Measured over the " "39-document corpus: 32 of the 32 documents with a plan lost " "their opening before this moved -- 159 704 characters, 9.18 " "%% of the corpus. It adds no boundary and moves one span's start. " "ON since 2026-09-10" ), ) build_parser.add_argument( "--no-first-span-from-zero", action="store_false", dest="first_span_from_zero", help=( "The repair's explicit opt-out, for a consumer who needs the " "pre-2026-09-10 first span byte for byte" ), ) build_parser.add_argument( "--close-span-gaps", action="store_true", default=DEFAULT_CLOSE_SPAN_GAPS, help=( "Close a concept's span against the next SURVIVING concept, and " "the last one against the end of the text, so a candidate removed " "after its neighbour's span was fixed does not take its text out " "of the plan. Measured over the 39-document corpus: it closes the " "43 631 characters -- 2.51 %% of the corpus over 8 of the 32 " "documents with a plan -- that round 7 left, both the 26 041 " "between entries and the 17 590 after the last. It adds no " "boundary and moves only spans' ends. ON since 2026-09-11" ), ) build_parser.add_argument( "--no-close-span-gaps", action="store_false", dest="close_span_gaps", help=( "The repair's explicit opt-out, for a consumer who needs the " "pre-2026-09-11 spans byte for byte" ), ) build_parser.add_argument( "--contents-name", action="store_true", default=DEFAULT_CONTENTS_NAME, help=( "Admit a title into a contents run only when a NAME survives " "stripping its trailing page number, so a run of data rows is not " "read as a contents listing and discarded. Measured over the " "39-document corpus: clause 1 dropped 68 candidates over 11 " "documents, of which 19 over 5 documents were data rows; the rule " "rescues 16 of the 19 and 0 of the 49 real contents entries. It " "only removes members from a run, so it can only add concepts. " "ON since 2026-09-09" ), ) build_parser.add_argument( "--no-contents-name", action="store_false", dest="contents_name", help=( "The repair's explicit opt-out, for a consumer who needs the " "pre-2026-09-09 contents-run predicate byte for byte" ), ) build_parser.add_argument( "--bold-title", action="store_true", default=DEFAULT_BOLD_TITLE, help=( "Read a line that is ONE bold span as a title, in a document that " "declares no heading of its own. For the type whose container has " "no heading style at all: `rtf` reached round 9 at 0 of 0 declared " "headings, 0 concepts and 1368 of 1368 characters in no segment. " "The grammar is markdown, so it reaches every type the converter " "writes bold for, and it is inert for `pdf`, which never goes " "through the converter. Measured over 47 readable documents: 0 " "false positives on the 31 that declare, by the gate" ), ) build_parser.add_argument( "--no-bold-title", action="store_false", dest="bold_title", help="The rule's explicit opt-out", ) build_parser.add_argument( "--pdf-headings", choices=("none", "font", "font-reserve"), default=( "font" if DEFAULT_PDF_HEADINGS else "font-reserve" if DEFAULT_PDF_HEADINGS_RESERVE else "none" ), help=( "how a PDF's headings are recovered before any arm reads the text. " "none (the default) is today's extraction: a PDF carries no heading " "markup and none is inferred. font infers one from TYPOGRAPHY -- a " "line whose dominant size is above the document's body median AND " "whose dominant font name says bold -- and emits it as an ATX " "heading in the same markdown the office path produces, so the " "existing heading rule reads it and no PDF-only grammar exists. A " "CONJUNCTION, measured: adding weight as a disjunct took precision " "from 0.786 to 0.524. font-reserve is font applied ONLY to a " "document whose own numbering the outline arm finds nothing of -- " "typography as a second source where there is no first one, never " "on top of one. Three values of one option, so no caller can ask " "for two at once" ), ) build_parser.add_argument( "--pdf-outline", action="store_true", default=DEFAULT_PDF_OUTLINE, help=( "OFF. Cut a PDF at the boundaries its own /Outlines bookmark tree " "declares, instead of at the ones the text rules recover. It is a " "SEGMENTATION arm and not a reader option: the extracted text is " "byte for byte the same either way, and a PDF that carries no " "bookmark tree builds byte-identically with the flag on. Measured " "on one 701-page process code whose publisher also ships a NISO-STS " "structure for it: the text rules recover 1967 of 2761 titled " "sections and 0 of its 28 chapters, while its bookmark tree matches " "2761 of 2761 exactly. The title comes from the BOOKMARK, so it is " "not cut short at the page's line break, and a page before the " "first bookmark destination is the table of contents rather than a " "second copy of the body. ONE document, ONE format, ONE publisher, " "and a bookmark tree is the publisher's CLAIM about its own " "structure -- a stale or wrong tree carries its error straight into " "the segmentation" ), ) build_parser.add_argument( "--no-pdf-outline", action="store_false", dest="pdf_outline", help="The arm's explicit opt-out", ) build_parser.add_argument( "--ocr", action="store_true", default=DEFAULT_OCR, help=( "read a PDF page as an IMAGE when its own text never arrived -- the " "page extracts empty, or as a page of (cid:N) placeholder codes. " "Requires the optional 'ocr' group (pip install " "'llm-ingestion-okf[extract,ocr]'); without it every affected file " "is a coded rejection, never a crash. Never a default and never " "able to become one: the engine is an optional dependency. OCR text " "is a reading of a picture, so it is deterministic within one model " "version and not across one" ), ) build_parser.add_argument("--report", type=Path, default=None, help="also write the report") return parser.parse_args(argv) def main(argv: list[str] | None = None) -> int: arguments = list(sys.argv[1:] if argv is None else argv) if arguments and arguments[0] in DELEGATED: return _delegate(arguments[0], arguments[1:]) args = parse_args(arguments) if not args.inbox.is_dir(): print(f"{CLI_ID}: FAILED - no such folder: {args.inbox}", file=sys.stderr) return 2 try: report = build( args.inbox, args.bundle, ingested_at=args.ingested_at, proposed_at=args.proposed_at, bundle_id=args.bundle_id, okf_version=args.okf_version, segments=args.segments == "on", plans_dir=args.plans_dir, okf_type=args.okf_type, outline_run=args.outline_run, table_grid=args.table_grid, unit_fold=args.unit_fold, keep_table_heading=args.keep_table_heading, sheet_section_rows=args.sheet_section_rows, drop_wrapped_outline=args.drop_wrapped_outline, outline_gate=args.outline_gate, first_span_from_zero=args.first_span_from_zero, close_span_gaps=args.close_span_gaps, contents_name=args.contents_name, bold_title=args.bold_title, pdf_headings=args.pdf_headings == "font", pdf_headings_reserve=args.pdf_headings == "font-reserve", ocr=args.ocr, pdf_outline=args.pdf_outline, ) except (IngestError, OSError, ValueError) as exc: print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr) return 2 if args.report is not None: args.report.parent.mkdir(parents=True, exist_ok=True) args.report.write_text(report.render(), encoding="utf-8", newline="") print(report.render()) if report.unaccounted or report.merged + report.rejected != report.n: print( f"{CLI_ID}: K1b FAILED - merged ({report.merged}) + coded rejections " f"({report.rejected}) != N ({report.n}). Unaccounted: " f"{', '.join(report.unaccounted) or '(none named)'}", file=sys.stderr, ) return 1 return 0 if __name__ == "__main__": raise SystemExit(main())