llm-ingestion-okf/docs/plan/office-intake.md

# Office intake: the three decisions, and what measured them

Decision record for the intake work planned 2026-09-01. The plan itself is
local-only; these three choices are not, because each of them changes something
a consumer can observe.

## 1. The converter binary is vendored, not resolved from PATH

**Decision: operator, 2026-09-01, after research.**

`pandoc` converts `docx`, `xlsx`, `pptx`, `odt` and `rtf` to text, and it enters
as a **pinned wheel inside the existing `[extract]` extra** rather than as a
system binary the library shells out to.

The first design considered was the opposite — spawn a system `pandoc` from one
allowlisted module — and three independent measurements retired it:

- The `xlsx` and `pptx` readers exist only from pandoc **3.8.3**. Debian 12
  ships 2.17.1.1 and Ubuntu 24.04 ships 3.1.3, so a `PATH` binary cannot deliver
  two of the five formats on current stable distributions. Those table rows
  would fail by environment rather than by design.
- The `docx` reader changed its emitted output in **at least 22 of the 3.x
  releases**, including one change shipped in 3.6.2 and reverted in 3.6.4. A
  byte-exact golden over an unpinned binary is a promise about someone else's
  machine.
- The ecosystem already made this decision: Quarto bundles pandoc to eliminate
  version-mismatch risk, and `pypandoc-binary` ships it as a platform wheel.

Two consequences follow, and both are load-bearing:

**The single-runtime-dependency rule is unchanged.** The wheel lives in the
`[extract]` extra, which `pyproject.toml` already places outside that rule by
construction. The default install still declares exactly one runtime dependency.

**The model-free gate keeps `subprocess` in its alternation.** Because the
conversion goes through an imported package rather than a process this library
spawns, `src/llm_ingestion_okf/` needs no `subprocess` of its own and the gate
stays exactly as it was. What the gate proves has narrowed and should be stated
honestly: it proves no model vendor is reachable from `src/`, not that no
process is spawned anywhere in the dependency tree.

**One measured trap, recorded because it is invisible when it fires.**
`pypandoc` searches `PATH` **before** its own bundled binary and selects the
highest version it finds. On a machine with a newer system pandoc, the vendored
wheel is silently bypassed and every determinism guarantee with it. The binary
must therefore be resolved explicitly and its version asserted against a frozen
literal — a check that costs nothing and is the only thing standing between the
pin and a bundle built by a converter nobody chose.

## 2. Judgement lives outside the run path, as versioned data

**Decision: operator, 2026-09-01.**

Segment boundaries are proposed by a stdlib, model-free tool; a `tools/`-side
command raises those proposals into reasoned candidates; a human ratifies or
corrects; the ratification is written to a versioned plan file, and the run path
only ever reads that file. `src/` remains model-free.

The measured shape of the loop matters more than the shape of the diagram:

- **The verdict carries its dwell time.** A ratified flag with no per-item time
  is unfalsifiable. The published case is 300 000+ insurance decisions averaging
  1.2 seconds each, ~80 % of the appealed ones later overturned. Recording the
  time is what makes "adjudicated" a fact rather than a claim, and it is the
  same field that measures what adjudication costs.
- **Rejecting is one keystroke; correcting is optional.** Measured over 2 784
  annotations: people flagged *fewer* errors when flagging required typing the
  correction. An interface that charges for disagreement manufactures the
  rubber-stamping its output then claims to have avoided.
- **The model leg is off by default,** so its value can be measured rather than
  assumed. Pre-annotation has been measured lowering a good annotator's accuracy
  from 98.1 % to 95.8 %.

## 3. A new profile, never a change to an existing one

**Decision: operator, 2026-09-01.**

A segmented bundle could not declare which upstream spec it targets:
`SEGMENTED_V1` names `bundle_id`, `OKF_V0_2` names `okf_version`, and the two
never intersect. Rather than add the key to `SEGMENTED_V1`, a sixth profile —
`SEGMENTED_OKF_V0_2` — carries both.

This follows the standing rule that upstream support is additive, expressed as a
new profile and never as a migration. It also avoids resting on an unverified
premise: extending `SEGMENTED_V1` would have been safe only if nobody had
adopted it, and "nobody has adopted it yet" is a claim about other repositories
that this one cannot check.

## What these three do not settle

The acceptance rubric these decisions are measured against (K1–K7) is owned at
programme level, spans this library and its consumers, and is not closed here.
This work produces numbers on K1–K5 with their denominators; the thresholds that
turn a number into a verdict belong to whoever owns the rubric.

## Two cross-repo decisions ratified after these three

The three above were this repository's own to make. The two below were not: they
span this library and a consumer, so they were ratified at programme level on
2026-09-02 and are recorded here because a contract that lives only in a
consumer's plan file is a contract this side cannot be held to. They enter the
office-intake work as amendments to it, not as new scope.

### 4. Concept IDs are bundle-local; identity across bundles is a tuple

**Decision: programme level, 2026-09-02.**

A concept ID is **bundle-local and stable**. Identity **across** bundles is the
tuple `(bundle_id, concept_id)` — never the concept ID alone. The producer
already stamps `bundle_id` into every concept's frontmatter, so a consumer reads
the whole tuple from a single document and needs the root `index.md` only as a
fallback for an older bundle.

**There is no cross-bundle link form in v0.2.** A consumer that holds several
bundles partitions per bundle; nothing in this library emits a syntax for
pointing from inside one bundle into another, and adding one is out of scope
until a consumer states the need.

The timing is the reason this is written down now rather than later. The moment a
golden bundle carrying `okf_version` is byte-pinned, the IDs inside it stop being
adjustable — retrofitting a different denotation afterwards would move a pinned
golden, which is the one thing byte-pinning exists to prevent.

### 5. The adjudication state has a fixed wire form

**Decision: programme level, 2026-09-02.**

Segment concepts carry their adjudication state in a frontmatter key
`adjudication`, whose value set is **closed**: `proposed` or `adjudicated`. A
value outside that set is an error, not an extension point.

When the value is `adjudicated`, three keys are written alongside it:

| Key | Form |
|---|---|
| `adjudicated_by` | the adjudicator's identifier |
| `adjudicated_at` | ISO 8601 |
| `adjudication_dwell_s` | integer seconds |

The state is projected into the index as the facet `adjudication`. All of it is
written **only under the new profile**; `SEGMENTED_V1` output stays byte-identical.

Two things make this a contract rather than a naming preference. First, the dwell
time travels with the verdict: a ratified flag with no per-item time is
unfalsifiable, and the same field is what makes adjudication throughput
measurable at all. Second, the consumer side has a rule this side must not
break — **absence of the key means the state `unknown` (an older bundle), never
`absent`**. A producer that emitted the key inconsistently would collapse that
distinction, and the consumer's ability to see an unratified concept and
discount it is the whole point of surfacing the state.