open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

test(fidelity): K2 on pptx, odt and rtf -- 59/59 and 20/20, on a set we wrote

Browse source 
`docs/2026-09-04-k2-pptx-odt-rtf.md` measured the corpus denominator for
these three rows at zero, so K2 for them was unmeasured rather than
passing. This measures them on the synthetic set committed in `a7b050b`,
whose hand count landed BEFORE any of these numbers existed.

Coverage 59/59 and pairing 20/20 for all three, standalone and product
alike, where "product" is scored off the persisted concept in the bundle
rather than an intermediate. The instrument's own denominator, derived
from each file's XML or stream, agrees with the hand count on all three:
59 strings, 20 pairable. Door B over the fixture directory persists 3/3
with nothing quarantined, rejected, failed or skipped, so K1b holds at
3 = N. There is no sub-100 % cell to explain, and that is a result about
three documents we wrote, not about the formats.

THE NEGATIVE CONTROL IS WHAT MAKES THOSE FIGURES MEAN ANYTHING. Removing
one label and its value from each converted text moves every row to
57/59 and 19/20 -- exactly the two strings and the one pair taken away.
`tests/test_fidelity.py` carries that control per reader, so it cannot
rot into a claim.

The order names `okf_fidelity` as the instrument and the instrument
refused all three types (`no source-string reader for '.pptx'`), so step
2 could not run without readers for them. Added here, stated in the
report as sitting outside the order's enumerated fence: this is the
measuring instrument, not extraction code. `extract.py` is untouched and
every `_EVIDENCE` row still reads `unmeasured` -- three synthetic
documents in one house style are exercise, not coverage, and the suite
asserts the rows are not promoted.

Two converter findings, reported and not fixed, both exit code 0 with no
warning. RTF cell paragraphs without `\pard\intbl` are read as each row
nested inside the previous one, 2076 characters where 117 were expected.
And the `\uN?` unicode escape -- THE FORM WORD EMITS -- loses the
character after it: `A\u248?BC` reads back as `AoC`. Any Word-produced
RTF carrying Norwegian letters loses one character after each of them,
silently. That is now a named limit of the `rtf` row rather than an
unknown.

Pandoc 3.9 and the host's 3.10.2 produce byte-identical output for all
three fixtures, so the numbers are not sitting on a version-fragile edge.
`~/corpora/` was neither read nor written; N there is still 43.

Suite 1141 passed, `ruff` clean, `mypy --strict src/ tools/` clean over
25 files, all after `git add`. Regenerating the fixtures leaves
`git diff --quiet` clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 05:22:12 +02:00
commit bd44929c89
3 changed files with 443 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

186
docs/2026-09-07-k2-pptx-odt-rtf-fixtures.md Normal file
View file

@ -0,0 +1,186 @@
# K2 for pptx/odt/rtf on a synthetic denominator, N = 3, 2026-09-07
Order `20260906T220349Z-652851369-from-.claude`. Closes the gap the previous
report opened rather than filled: `docs/2026-09-04-k2-pptx-odt-rtf.md` measured
the corpus denominator for these three office rows and found it **zero** —
`~/corpora/okf-telling-20260829/K2/trinn1` holds 43 files and not one is a
`pptx`, an `odt` or an `rtf`, with a known-positive control (`docx` → 5/5)
proving the query could find. So K2 for those rows was *unmeasured*, which is a
different fact from *passes*, and the corpus could not be made to say otherwise.
This report measures them on a **synthetic** denominator instead: three
documents written here, with a hand count committed before the measurement ran.
That is a weaker instrument than a corpus and the limits are stated in full at
the bottom. It is not weaker than nothing, which is what the row had.
**Nothing in `~/corpora/` was read or written.** N there is still 43.
## What was measured, and against what
`tools/okf_fidelity.py` asks the source document's own two questions:
- **coverage** — of the strings the FILE stores, how many appear in the
converted text. Read from the package's own XML (or, for RTF, from the
stream), never from a converter's output: a fasit derived from one converter
would score that converter on its own homework.
- **pairing** — of the rows carrying a label and a value, how many keep both on
one output line. That is the criterion a requirement table is read by.
The fixture set is `tests/fixtures/k2-office/`: **one authored document laid out
three times, in three containers**, so the container and its reader are the only
variable between the three measurements. A title, an intro, a 20-row
label/value table, a caption, and a 4×4 grid. Norwegian throughout, with the
diacritics the pipeline has to survive.
**The fasit was hand-counted from the authored content and committed first**, in
`a7b050b`, before any of the numbers below existed:
| | count | how it was counted |
|---|---|---|
| cells | **56** | 20 rows × 2 columns + 4 rows × 4 columns |
| pairs | **20** | the 20 label rows; every label ends in a colon |
| distinct strings | **59** | title + intro + caption + 56 cells, all distinct after normalisation |
The instrument derives its own denominator from each file's XML or stream,
independently of that hand count. **The two agree on all three documents:
59 strings, 20 pairable.** A disagreement would have been the report; there
isn't one.
## The result
Converter: the **vendored pandoc 3.9**, resolved by path and version-asserted by
`src/llm_ingestion_okf/_pandoc.py`, with `-t markdown --eol=lf --wrap=none`.
"Product" is the text scored off the **persisted concept in the bundle** — the
artifact Door B actually writes, not an intermediate.
| format | coverage | paired | denominator |
|---|---|---|---|
| `pptx` | **59/59 (100 %)** | **20/20 (100 %)** | 59 strings, 20 pairs |
| `odt` | **59/59 (100 %)** | **20/20 (100 %)** | 59 strings, 20 pairs |
| `rtf` | **59/59 (100 %)** | **20/20 (100 %)** | 59 strings, 20 pairs |
Standalone (converter alone) and product (through `extract_text`, and through
the written concept) are equal for all three — no fall from one to the other, so
there is nothing here to attribute to this library.
**There is no sub-100 % cell to explain.** The order asks that every fall below
100 % be named with a cause; there are none, and that absence is a result about
a document set we wrote, not about the format.
**Door B, product path, N = 3:** `process_inbox` over the fixture directory
persisted **3/3**, with 0 quarantined, 0 rejected, 0 failed and 0 skipped.
**K1b: 3 + 0 = 3 = N.**
### The negative control
100 % means nothing from an instrument that cannot see a loss. Removing one
label and its value from each converted text, and re-scoring the same shipped
fixtures:
| format | coverage | paired |
|---|---|---|
| `pptx` | 57/59 (96.6 %) | 19/20 (95.0 %) |
| `odt` | 57/59 (96.6 %) | 19/20 (95.0 %) |
| `rtf` | 57/59 (96.6 %) | 19/20 (95.0 %) |
The instrument moves by exactly the two strings and the one pair that were
taken away. `tests/test_fidelity.py` carries the same control per reader as a
test, so it cannot rot.
### Converter version
The order asks for a comparison against the host's newer converter if the
numbers fell. They did not, and the comparison was run anyway because it is
cheap and it bounds a different risk: **pandoc 3.9 and the host's pandoc 3.10.2
produce byte-identical output for all three fixtures** (`cmp`, three files,
no differences). These numbers are therefore not sitting on a version-fragile
edge.
## Two converter findings, neither of them fixed
Both were measured while laying the RTF out, both are structurally plausible
input read silently wrong, and both come back **exit code 0 with no warning** —
the same shape as the missing `word/styles.xml` and the `inlineStr` xlsx already
recorded in `tests/fixtures/README.md`. Per the order they are reported, not
patched; nothing in `src/` works around either.
**1. RTF cell paragraphs need `\pard\intbl`.** Without it, consecutive
`\trowd…\row` rows are read as each row NESTED inside the previous one. Five
label/value rows came back as five levels of nested table — **2076 characters
where 117 were expected**. The fixture writes `\intbl`, which is correct RTF; a
document that omits it is not this library's problem to fix, but it is a shape a
real drop directory can contain.
**2. The `\uN?` unicode escape loses the character after it.** This is the form
**Word emits**, so it is the common case rather than an edge one. Measured
directly against the vendored 3.9:
```
A\u248?BC -> AøC the ring letter arrives, the B is GONE
A\u248?xBC -> AøBC the same rule seen from the other side
A\u248 ?BC -> AøBC an explicit delimiter, and the text survives
```
The `?` is taken as the control word's delimiter, and `\uc1` then skips a *real*
character. `\uc0` does not change it. The fixture is written in the form that
round-trips (`\uN ?`), so this finding is recorded here rather than baked
invisibly into a fixture.
**What this means for a real `rtf` drop.** Any Word-produced RTF containing
Norwegian letters will lose one character after each of them, silently. That is
a defect in the converter, not in this library, and it is now a named limit of
the `rtf` row rather than an unknown.
## Honesty limits
Read these as part of the table, not as a footnote to it.
- **Synthetic, N = 3, one house style.** Three documents we wrote are not a
corpus. They are internally consistent by construction: one author, one
layout convention, one set of table shapes. A real `pptx` deck with text
boxes over images, an `odt` with tracked changes, or an RTF from a different
producer are all outside what was measured.
- **We authored the fixtures to be readable, and they read.** The 100 % says
the path works end to end on documents built to exercise it. It does not say
the format is covered. `extract._EVIDENCE` therefore still reads
`unmeasured` for all three rows, and `tests/test_k2_office_fixtures.py`
asserts that it does — so a later reader cannot mistake this report for
promotion.
- **The `rtf` row is the weakest of the three.** For `pptx` and `odt` the fasit
is read out of a package format that neither we nor the converter defined.
For `rtf` there is no part to read, so the fasit comes from a reader written
here against a stream written here. Both halves are ours, and the
independence the other two rows have is missing.
- **Drawn content is out of scope entirely.** Conversion recovers text; a
figure, a diagram or a drawn shape has no text to recover, only a caption.
Every office extraction already warns about this on every call. The fixtures
contain no drawn content, so this report measures nothing about it.
- **`pptx` speaker notes, `odt` footnotes and RTF headers/footers** carry text
in real documents and appear in none of these fixtures. Unmeasured, and named
here so the gap has a denominator of its own.
## What changed in the tree
- `tests/fixtures/k2-office/` — the three documents, hand-laid by
`tests/fixtures/make_k2_office.py`. **No converter wrote any of them**: a file
written by the converter and read by the converter proves only that the
converter agrees with itself. The commissioning order offered pandoc as one
generator option; the committed fixture policy in `tests/fixtures/README.md`
forbids it, and the policy wins.
- `tests/fixtures/k2-office-fasit.json` — the hand count, committed first.
- `tests/test_k2_office_fixtures.py` — pins N = 3, 3/3 through Door B offline,
the concept contents, the fasit, and that the rows stay `unmeasured`.
- `tools/okf_fidelity.py` — **source-string readers for `pptx`, `odt` and
`rtf`.** Stated plainly because it sits outside the order's enumerated fence:
the instrument the order names refused all three types
(`ValueError: no source-string reader for '.pptx'`), so step 2 could not run
without it. It is the measuring instrument, not extraction code —
`src/llm_ingestion_okf/extract.py` is untouched, and so is every `_EVIDENCE`
row.
**Suite: 1141 passed** (1127 before this work, 1132 after the fixture commit,
1141 with the reader tests). `ruff check`, `ruff format --check` and
`mypy --strict src/ tools/` (25 files) clean, all measured after `git add`.
Re-running `make_k2_office.py` leaves `git diff --quiet` clean on
`tests/fixtures/k2-office`, so the three documents are reproducible from that
one file. Nothing pushed.