open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

test(fixtures): a synthetic K2 denominator for pptx, odt and rtf

Browse source 
`docs/2026-09-04-k2-pptx-odt-rtf.md` measured the corpus denominator for
these three office rows and found it ZERO: `K2/trinn1` holds 43 files and
not one of them is a `pptx`, an `odt` or an `rtf`. So `extract._EVIDENCE`
calls those rows `unmeasured` in the strongest sense available -- they
work by construction and had never met a document at all.

This is the smallest thing that changes that without inventing a corpus.
One authored document -- a title, an intro, a 20-row label/value table, a
caption and a 4x4 grid -- laid out three times in three containers, so
the container and its reader are the only variable between the three
measurements. `k2-office-fasit.json` carries the hand count taken from
the AUTHORED content rather than from any converter's output: 56 cells,
20 pairs, 59 distinct strings, shared by all three. It is committed here,
before the measurement runs, because a fasit written afterwards is a
description of a result rather than a denominator for it.

No converter wrote any of these files. `make_k2_office.py` lays every
part by hand, for the reason `make_fixtures.py` already states and this
set inherits: a file written by the converter and then read by the
converter proves only that the converter agrees with itself, and stays
green through any conversion defect that is symmetric. The commissioning
order offered pandoc as one generator option; the committed fixture
policy forbids it and the policy wins.

Two converter behaviours were measured while laying the RTF out, both of
them structurally plausible input read silently wrong, exit code 0 and no
warning. Without `\pard\intbl` on cell paragraphs, consecutive
`\trowd...\row` rows come back as each row NESTED inside the previous
one: five label/value rows read as five levels of nested table, 2076
characters where 117 were expected. And the `\uN?` unicode escape -- the
form Word emits -- loses the character after it: `A\u248?BC` reads back
as `AoC` with the `B` gone, `A\u248?xBC` reads back as `AoBC`. The
fixture writes `\uN ?` with an explicit space, which round-trips. Neither
is worked around anywhere in `src/`.

The generator and the fasit live one level above `k2-office/` and that is
not tidiness: Door B walks its drop directory recursively, so anything
parked beside the three documents would enter the run and N would stop
being 3.

Three synthetic documents in one house style are not a corpus. The rows
stay `unmeasured` and the suite asserts that they do.

Suite 1132 passed (1127 + 5), `ruff check` and `ruff format --check`
clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 05:17:18 +02:00
commit a7b050b569
7 changed files with 593 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

50
tests/fixtures/README.md vendored
View file

@ -47,6 +47,56 @@ structurally valid input, silently reduced output, exit code 0 and no warning:
the xlsx extracts with the sheet name intact and **every cell value gone**.
The fixture therefore uses a `dimension` element and a shared string table.
## The K2 office fixture set (`k2-office/`)
`krav-presentasjon.pptx`, `krav-tekstdokument.odt` and
`krav-rikt-tekstformat.rtf` are the synthetic denominator for the three office
rows the corpus has none of. `docs/2026-09-04-k2-pptx-odt-rtf.md` measured that
denominator at **zero** — `K2/trinn1` holds 43 files and not one is a `pptx`,
an `odt` or an `rtf` — so those rows were `unmeasured` in the sense of never
having met a document at all. Regenerated by `make_k2_office.py` in this
directory:
```
python3 tests/fixtures/make_k2_office.py
```
**One document, three containers.** All three carry the same authored content —
a title, an intro, a 20-row label/value table, a caption and a 4x4 grid — so
the only variable between the three measurements is the container and the
reader that opens it. The counts are hand-counted once, in
`k2-office-fasit.json`, and shared: **56 cells, 20 pairs, 59 distinct strings.**
**The generator and the fasit live one level up, and that is not tidiness.**
Door B walks its drop directory recursively, so anything parked inside
`k2-office/` would enter the run and N would stop being 3.
Same policy as the office fixtures above, for the same reason: every part is
hand-laid and no converter wrote any of them. The commissioning order offered
pandoc as a generator option; a file written by the converter and then read by
the converter would prove only that the converter agrees with itself.
Two things were measured while building this set, both against the vendored
pandoc 3.9, and both are the house shape — structurally plausible input,
silently wrong output, exit code 0 and no warning:
- **RTF cell paragraphs need `\pard\intbl`.** Without it, consecutive
`\trowd…\row` rows are read as each row NESTED inside the previous one:
five label/value rows came back as five levels of nested table, 2076
characters where 117 were expected.
- **The `\uN?` unicode escape loses the character after it.** Measured
directly: `A\u248?BC` reads back as `AoC` (ring letter present, `B` gone) and
`A\u248?xBC` reads back as `AoBC`. The `?` is taken as the control word's
delimiter and `\uc1` then skips a real character. The fixture writes
`\uN ?` with an explicit space, which round-trips. **This is the form Word
emits**, so it is a converter finding rather than a fixture quirk — recorded
in `docs/2026-09-07-k2-pptx-odt-rtf-fixtures.md`, not worked around anywhere
in `src/`.
**Three synthetic documents in one house style are not a corpus.** The rows stay
`unmeasured` in `extract._EVIDENCE` and `tests/test_k2_office_fixtures.py`
asserts that they do.
## The proposer's default-profile golden
`propose-golden-default.json` is the artifact `tools/okf_propose_segments.py`

42
tests/fixtures/k2-office-fasit.json vendored Normal file
View file

@ -0,0 +1,42 @@
{
"_note": [
"Hand-counted from the AUTHORED content in make_k2_office.py, never from a",
"converter's output: a fasit read off one converter would score that",
"converter on its own homework. Committed before the measurement in",
"docs/2026-09-07-k2-pptx-odt-rtf-fixtures.md was run.",
"The three documents carry the SAME content in three containers, so the",
"counts are shared and the container is the only variable."
],
"counted_by": "hand, from tests/fixtures/make_k2_office.py",
"counted_at": "2026-09-07",
"shared": {
"cells": 56,
"pairs": 20,
"strings": 59
},
"how_counted": {
"cells": "20 rows x 2 columns (PAIRS) + 4 rows x 4 columns (GRID) = 40 + 16",
"pairs": "the 20 PAIRS rows; every label ends in a colon, which is the rule label_value_rows selects on",
"strings": "TITLE + INTRO + GRID_CAPTION + 56 cells = 59, all distinct after normalisation and all at or above MIN_STRING"
},
"documents": {
"krav-presentasjon.pptx": {
"cells": 56,
"pairs": 20,
"strings": 59,
"container": "two slides: title, intro and the 20-row table on slide 1; caption and the 4x4 grid on slide 2"
},
"krav-tekstdokument.odt": {
"cells": 56,
"pairs": 20,
"strings": 59,
"container": "one text body: heading, intro paragraph, 20-row table, caption paragraph, 4x4 table"
},
"krav-rikt-tekstformat.rtf": {
"cells": 56,
"pairs": 20,
"strings": 59,
"container": "flat stream: title paragraph, intro paragraph, 20 table rows, caption paragraph, 4 table rows"
}
}
}

BIN
tests/fixtures/k2-office/krav-presentasjon.pptx vendored Normal file
View file

Binary file not shown.

52
tests/fixtures/k2-office/krav-rikt-tekstformat.rtf vendored Normal file
View file

@ -0,0 +1,52 @@
{\rtf1\ansi\ansicpg1252\deff0{\fonttbl{\f0\froman Times New Roman;}}\pard\sa180\b\fs32 Kravspesifikasjon for tunnelbelysning\b0\fs24\par
\pard\sa180 Dokumentet samler kravene til belysning i vegtunneler over 500 meter.\par
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Dokumentnummer:\cell \pard\intbl SVV-2026-0417\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Tittel:\cell \pard\intbl Tunnelbelysning i hovedl\u248 ?pet\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Ansvarlig avdeling:\cell \pard\intbl Utbyggingsdivisjonen\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Fagomr\u229 ?de:\cell \pard\intbl Elektro og belysning\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Versjon:\cell \pard\intbl 2.3 godkjent\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Gyldig fra:\cell \pard\intbl 01.03.2026\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Erstatter:\cell \pard\intbl SVV-2024-0188\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Tunnellengde:\cell \pard\intbl 1240 meter\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Dimensjonerende fart:\cell \pard\intbl 80 km/t\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl \u197 ?rsd\u248 ?gntrafikk:\cell \pard\intbl 12400 kj\u248 ?ret\u248 ?y\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Terskelluminans:\cell \pard\intbl 145 candela\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Overgangssone:\cell \pard\intbl Tre trinn nedtrapping\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Innerstrekning:\cell \pard\intbl 3,0 candela\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Utkj\u248 ?ringssone:\cell \pard\intbl Ingen forsterkning\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Fargetemperatur:\cell \pard\intbl 4000 kelvin\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Fargegjengivelse:\cell \pard\intbl Ra st\u248 ?rre enn 70\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl N\u248 ?dbelysning:\cell \pard\intbl 60 minutter drift\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Vedlikeholdsfaktor:\cell \pard\intbl 0,80 beregnet\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl M\u229 ?lemetode:\cell \pard\intbl M\u229 ?ling med luminanskamera\cell \row
\trowd\trgaph108\cellx3000\cellx6000
\pard\intbl Avviksh\u229 ?ndtering:\cell \pard\intbl S\u248 ?knad om fravik\cell \row
\pard\sa180 Luminansmatrise per trafikklasse\par
\trowd\trgaph108\cellx3000\cellx6000\cellx9000\cellx12000
\pard\intbl Sone\cell \pard\intbl Klasse A\cell \pard\intbl Klasse B\cell \pard\intbl Klasse C\cell \row
\trowd\trgaph108\cellx3000\cellx6000\cellx9000\cellx12000
\pard\intbl Terskel\cell \pard\intbl 150 cd\cell \pard\intbl 120 cd\cell \pard\intbl 95 cd\cell \row
\trowd\trgaph108\cellx3000\cellx6000\cellx9000\cellx12000
\pard\intbl Overgang\cell \pard\intbl 45 cd\cell \pard\intbl 36 cd\cell \pard\intbl 28 cd\cell \row
\trowd\trgaph108\cellx3000\cellx6000\cellx9000\cellx12000
\pard\intbl Indre\cell \pard\intbl 3,5 cd\cell \pard\intbl 3,0 cd\cell \pard\intbl 2,5 cd\cell \row
}

BIN
tests/fixtures/k2-office/krav-tekstdokument.odt vendored Normal file
View file

Binary file not shown.

333
tests/fixtures/make_k2_office.py vendored Normal file
View file

@ -0,0 +1,333 @@
"""Regenerate the K2 office fixture set: one pptx, one odt, one rtf.
Three containers carrying the SAME document, so the only variable between the
three measurements is the container and the reader that opens it. Hand-laid,
part by part, with no generator library anywhere -- for the reason
`make_fixtures.py` states and this set inherits: a file written by the
converter and then read by the converter proves only that the converter agrees
with itself, and would stay green through any conversion defect that is
symmetric, which is most of them. The order that commissioned this set offered
pandoc as one generator option; this repository's committed fixture policy
forbids it, and the policy wins.
The document is invented, and that is a constraint rather than a detail: a
plausible Norwegian requirements sheet carrying the diacritics the pipeline has
to survive, and nobody's real document. No private file, nothing from
`~/Documents`, nothing from a customer.
Run from the repository root: python3 tests/fixtures/make_k2_office.py
"""
from __future__ import annotations
import io
import zipfile
from pathlib import Path
HERE = Path(__file__).parent
OUT = HERE / "k2-office"
TITLE = "Kravspesifikasjon for tunnelbelysning"
INTRO = "Dokumentet samler kravene til belysning i vegtunneler over 500 meter."
# The pairing table. Every label ends in a colon, which is the rule
# `okf_fidelity.label_value_rows` selects on -- so these twenty rows ARE the
# pairable denominator, and a converter that drops a value or breaks a row over
# two output lines shows up as a fall from 20.
PAIRS: tuple[tuple[str, str], ...] = (
("Dokumentnummer:", "SVV-2026-0417"),
("Tittel:", "Tunnelbelysning i hovedløpet"),
("Ansvarlig avdeling:", "Utbyggingsdivisjonen"),
("Fagområde:", "Elektro og belysning"),
("Versjon:", "2.3 godkjent"),
("Gyldig fra:", "01.03.2026"),
("Erstatter:", "SVV-2024-0188"),
("Tunnellengde:", "1240 meter"),
("Dimensjonerende fart:", "80 km/t"),
("Årsdøgntrafikk:", "12400 kjøretøy"),
("Terskelluminans:", "145 candela"),
("Overgangssone:", "Tre trinn nedtrapping"),
("Innerstrekning:", "3,0 candela"),
("Utkjøringssone:", "Ingen forsterkning"),
("Fargetemperatur:", "4000 kelvin"),
("Fargegjengivelse:", "Ra større enn 70"),
("Nødbelysning:", "60 minutter drift"),
("Vedlikeholdsfaktor:", "0,80 beregnet"),
("Målemetode:", "Måling med luminanskamera"),
("Avvikshåndtering:", "Søknad om fravik"),
)
# The 4x4 the order asks for, and a second question. No cell here ends in a
# colon, so the grid contributes to COVERAGE and to nothing else: a converter
# that flattens a grid into prose keeps its coverage and loses its shape, and
# the two numbers are meant to be able to move independently.
# The caption between the two tables. A PARAGRAPH, not an empty one: an
# empty `\\pard\\par` left the converter joining the 2-column rows and the
# 4-column rows into a single four-column table, so the grid stopped being a
# second table at all.
GRID_CAPTION = "Luminansmatrise per trafikklasse"
GRID: tuple[tuple[str, ...], ...] = (
("Sone", "Klasse A", "Klasse B", "Klasse C"),
("Terskel", "150 cd", "120 cd", "95 cd"),
("Overgang", "45 cd", "36 cd", "28 cd"),
("Indre", "3,5 cd", "3,0 cd", "2,5 cd"),
)
def _escape(text: str) -> str:
return text.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
_ZIP_DATE = (2020, 1, 1, 0, 0, 0)
_XML = '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
def build_container(parts: dict[str, str], *, stored_first: str | None = None) -> bytes:
"""Zip the parts with a fixed timestamp, so the fixture is byte-reproducible.
A zip records mtime, so without a constant `date_time` the fixture would
differ on every regeneration and `git diff --quiet` could not be the check.
`stored_first` exists for ODF, whose specification requires `mimetype` to be
the first member and stored uncompressed.
"""
out = io.BytesIO()
with zipfile.ZipFile(out, "w", compression=zipfile.ZIP_DEFLATED) as archive:
names = list(parts)
if stored_first is not None:
names.remove(stored_first)
info = zipfile.ZipInfo(stored_first, date_time=_ZIP_DATE)
info.compress_type = zipfile.ZIP_STORED
archive.writestr(info, parts[stored_first])
for name in names:
info = zipfile.ZipInfo(name, date_time=_ZIP_DATE)
info.compress_type = zipfile.ZIP_DEFLATED
archive.writestr(info, parts[name])
return out.getvalue()
# --- odt ---------------------------------------------------------------------
_ODT_NS = (
' xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0"'
' xmlns:text="urn:oasis:names:tc:opendocument:xmlns:text:1.0"'
' xmlns:table="urn:oasis:names:tc:opendocument:xmlns:table:1.0"'
' xmlns:style="urn:oasis:names:tc:opendocument:xmlns:style:1.0"'
' xmlns:fo="urn:oasis:names:tc:opendocument:xmlns:xsl-fo-compatible:1.0"'
' office:version="1.3"'
)
def _odt_table(name: str, rows: tuple[tuple[str, ...], ...]) -> str:
columns = len(rows[0])
body = "".join(
"<table:table-row>"
+ "".join(
f'<table:table-cell office:value-type="string">'
f"<text:p>{_escape(cell)}</text:p></table:table-cell>"
for cell in row
)
+ "</table:table-row>"
for row in rows
)
return (
f'<table:table table:name="{name}">'
f'<table:table-column table:number-columns-repeated="{columns}"/>'
f"{body}</table:table>"
)
def odt_parts() -> dict[str, str]:
content = (
f"{_XML}<office:document-content{_ODT_NS}><office:body><office:text>"
f'<text:h text:outline-level="1">{_escape(TITLE)}</text:h>'
f"<text:p>{_escape(INTRO)}</text:p>"
+ _odt_table("Krav", tuple(PAIRS))
+ f"<text:p>{_escape(GRID_CAPTION)}</text:p>"
+ _odt_table("Luminansmatrise", GRID)
+ "</office:text></office:body></office:document-content>"
)
return {
"mimetype": "application/vnd.oasis.opendocument.text",
"META-INF/manifest.xml": _XML
+ '<manifest:manifest xmlns:manifest="urn:oasis:names:tc:opendocument:xmlns:manifest:1.0"'
+ ' manifest:version="1.3">'
+ '<manifest:file-entry manifest:full-path="/"'
+ ' manifest:media-type="application/vnd.oasis.opendocument.text"/>'
+ '<manifest:file-entry manifest:full-path="content.xml"'
+ ' manifest:media-type="text/xml"/>'
+ '<manifest:file-entry manifest:full-path="styles.xml"'
+ ' manifest:media-type="text/xml"/>'
+ "</manifest:manifest>",
"styles.xml": _XML
+ f"<office:document-styles{_ODT_NS}><office:styles/></office:document-styles>",
"content.xml": content,
}
# --- pptx --------------------------------------------------------------------
_A = "http://schemas.openxmlformats.org/drawingml/2006/main"
_P = "http://schemas.openxmlformats.org/presentationml/2006/main"
_R = "http://schemas.openxmlformats.org/officeDocument/2006/relationships"
def _pptx_text_body(text: str) -> str:
return "<p:txBody><a:bodyPr/><a:p><a:r><a:t>" + _escape(text) + "</a:t></a:r></a:p></p:txBody>"
def _pptx_shape(shape_id: int, name: str, text: str) -> str:
return (
"<p:sp><p:nvSpPr>"
f'<p:cNvPr id="{shape_id}" name="{_escape(name)}"/><p:cNvSpPr/><p:nvPr/>'
"</p:nvSpPr><p:spPr/>" + _pptx_text_body(text) + "</p:sp>"
)
def _pptx_table(shape_id: int, name: str, rows: tuple[tuple[str, ...], ...]) -> str:
columns = len(rows[0])
grid = "".join('<a:gridCol w="2000000"/>' for _ in range(columns))
body = "".join(
'<a:tr h="370840">'
+ "".join(
"<a:tc><a:txBody><a:bodyPr/><a:p><a:r><a:t>"
+ _escape(cell)
+ "</a:t></a:r></a:p></a:txBody><a:tcPr/></a:tc>"
for cell in row
)
+ "</a:tr>"
for row in rows
)
return (
"<p:graphicFrame><p:nvGraphicFramePr>"
f'<p:cNvPr id="{shape_id}" name="{_escape(name)}"/>'
"<p:cNvGraphicFramePr/><p:nvPr/></p:nvGraphicFramePr>"
'<p:xfrm><a:off x="0" y="0"/><a:ext cx="8000000" cy="4000000"/></p:xfrm>'
f'<a:graphic><a:graphicData uri="{_A}/table">'
f"<a:tbl><a:tblPr/><a:tblGrid>{grid}</a:tblGrid>{body}</a:tbl>"
"</a:graphicData></a:graphic></p:graphicFrame>"
)
def _pptx_slide(shapes: str) -> str:
return (
f'{_XML}<p:sld xmlns:a="{_A}" xmlns:r="{_R}" xmlns:p="{_P}">'
"<p:cSld><p:spTree>"
'<p:nvGrpSpPr><p:cNvPr id="1" name=""/><p:cNvGrpSpPr/><p:nvPr/></p:nvGrpSpPr>'
"<p:grpSpPr/>" + shapes + "</p:spTree></p:cSld></p:sld>"
)
def pptx_parts() -> dict[str, str]:
slide_one = _pptx_slide(
_pptx_shape(2, "Tittel", TITLE)
+ _pptx_shape(3, "Ingress", INTRO)
+ _pptx_table(4, "Kravtabell", tuple(PAIRS))
)
slide_two = _pptx_slide(
_pptx_shape(2, "Undertittel", GRID_CAPTION) + _pptx_table(3, "Luminansmatrise", GRID)
)
rels = (
f'{_XML}<Relationships xmlns="{_R}">'
'<Relationship Id="rId1" Type="{t}/slide" Target="slides/slide1.xml"/>'
'<Relationship Id="rId2" Type="{t}/slide" Target="slides/slide2.xml"/>'
"</Relationships>"
).format(t=_R)
return {
"[Content_Types].xml": _XML
+ '<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">'
+ '<Default Extension="xml" ContentType="application/xml"/>'
+ '<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>'
+ '<Override PartName="/ppt/presentation.xml" ContentType="application/vnd.openxmlformats-officedocument.presentationml.presentation.main+xml"/>'
+ '<Override PartName="/ppt/slides/slide1.xml" ContentType="application/vnd.openxmlformats-officedocument.presentationml.slide+xml"/>'
+ '<Override PartName="/ppt/slides/slide2.xml" ContentType="application/vnd.openxmlformats-officedocument.presentationml.slide+xml"/>'
+ "</Types>",
"_rels/.rels": _XML
+ f'<Relationships xmlns="{_R}">'
+ f'<Relationship Id="rId1" Type="{_R}/officeDocument" Target="ppt/presentation.xml"/>'
+ "</Relationships>",
"ppt/_rels/presentation.xml.rels": rels,
"ppt/presentation.xml": _XML
+ f'<p:presentation xmlns:a="{_A}" xmlns:r="{_R}" xmlns:p="{_P}">'
+ '<p:sldIdLst><p:sldId id="256" r:id="rId1"/><p:sldId id="257" r:id="rId2"/></p:sldIdLst>'
+ "</p:presentation>",
"ppt/slides/slide1.xml": slide_one,
"ppt/slides/slide2.xml": slide_two,
"ppt/slides/_rels/slide1.xml.rels": _XML + f'<Relationships xmlns="{_R}"/>',
"ppt/slides/_rels/slide2.xml.rels": _XML + f'<Relationships xmlns="{_R}"/>',
}
# --- rtf ---------------------------------------------------------------------
#
# RTF carries no unicode of its own: a character above the code page is written
# as an escape with an ASCII replacement beside it, for a reader too old to
# understand the escape. Emitting the letters raw would make the fixture's bytes
# depend on a code page nobody declared.
#
# THE SPACE IN `\\uN ?` IS LOAD-BEARING AND IT IS NOT COSMETIC. The form Word
# emits is `\\uN?` with no delimiter, and the vendored converter reads that as
# the control word delimited BY the `?`, then applies `\\uc1` to the character
# AFTER it. Measured directly: `A\\u248?BC` comes back as `AoC` with the ring
# letter in place and the `B` GONE; `A\\u248?xBC` comes back as `AoBC`, which
# is the same rule seen from the other side. An explicit space delimits the
# control word, so the `?` is what gets skipped and the text survives. The
# fixture is therefore written in the form that round-trips, and the form that
# does not is recorded in `docs/2026-09-07-k2-pptx-odt-rtf-fixtures.md` as a
# converter finding rather than worked around in silence.
_RTF_CELL_WIDTH = 3000
def _rtf_escape(text: str) -> str:
out = []
for char in text:
if char in "\\{}":
out.append("\\" + char)
elif ord(char) < 128:
out.append(char)
else:
out.append(f"\\u{ord(char)} ?")
return "".join(out)
def _rtf_row(row: tuple[str, ...]) -> str:
"""One table row. `\\pard\\intbl` PER CELL, and it is load-bearing.
Measured against the vendored converter while building this fixture: the
same rows WITHOUT `\\intbl` are read as each row nested inside the previous
one -- five label/value rows came back as five levels of nested table and
2076 characters where 117 were expected, exit code 0 and no warning. That
is the same shape as the missing `word/styles.xml` and the `inlineStr`
xlsx recorded in `make_fixtures.py`: structurally plausible input, silently
wrong output, nothing anywhere saying so.
"""
borders = "".join(f"\\cellx{_RTF_CELL_WIDTH * (index + 1)}" for index in range(len(row)))
cells = "".join(f"\\pard\\intbl {_rtf_escape(cell)}\\cell " for cell in row)
return "\\trowd\\trgaph108" + borders + "\n" + cells + "\\row\n"
def rtf_bytes() -> bytes:
body = [
"{\\rtf1\\ansi\\ansicpg1252\\deff0",
"{\\fonttbl{\\f0\\froman Times New Roman;}}",
"\\pard\\sa180\\b\\fs32 " + _rtf_escape(TITLE) + "\\b0\\fs24\\par\n",
"\\pard\\sa180 " + _rtf_escape(INTRO) + "\\par\n",
]
body.extend(_rtf_row(row) for row in PAIRS)
body.append("\\pard\\sa180 " + _rtf_escape(GRID_CAPTION) + "\\par\n")
body.extend(_rtf_row(row) for row in GRID)
body.append("}")
return "".join(body).encode("ascii")
if __name__ == "__main__":
OUT.mkdir(parents=True, exist_ok=True)
written = {
"krav-presentasjon.pptx": build_container(pptx_parts()),
"krav-tekstdokument.odt": build_container(odt_parts(), stored_first="mimetype"),
"krav-rikt-tekstformat.rtf": rtf_bytes(),
}
for name, payload in sorted(written.items()):
(OUT / name).write_bytes(payload)
print(f"wrote {name} ({len(payload)} bytes)")

116
tests/test_k2_office_fixtures.py Normal file
View file

@ -0,0 +1,116 @@
"""The K2 denominator for `pptx`/`odt`/`rtf`: a synthetic fixture set, N = 3.
`docs/2026-09-04-k2-pptx-odt-rtf.md` measured the corpus denominator for these
three office rows and found it ZERO -- `K2/trinn1` holds 43 files and not one
of them is a `pptx`, an `odt` or an `rtf`. So those three rows in
`extract._EVIDENCE` read `unmeasured`: they work by construction and had never
met a document anyone wrote.
This module is the smallest thing that changes that without inventing a corpus:
three hand-laid documents carrying the SAME content in three containers, and a
hand-counted fasit committed beside them. It pins the one property a report
cannot assert for itself -- that all three go through Door B, offline, and land
as concepts -- so the numbers in
`docs/2026-09-07-k2-pptx-odt-rtf-fixtures.md` stay re-measurable rather than
becoming a memory of a run.
WHAT IT DOES NOT DO. Three synthetic documents in one house style are not a
corpus. A green run here says the row is exercised, never that the format is
covered; `_EVIDENCE` therefore stays `unmeasured` and this suite asserts that
it does.
"""
from __future__ import annotations
import json
from dataclasses import dataclass
from pathlib import Path
import pytest
from llm_ingestion_okf.extract import _EVIDENCE
from llm_ingestion_okf.inbox import GateDecision, process_inbox
pytest.importorskip("pypandoc", reason="office conversion needs the [extract] extra")
INGESTED_AT = "2026-09-07T00:00:00Z"
WARN = "warn"
FIXTURES = Path(__file__).parent / "fixtures"
DROP = FIXTURES / "k2-office"
FASIT = FIXTURES / "k2-office-fasit.json"
#: The three containers, one per row this fixture set exists to reach.
EXPECTED_FILES = (
"krav-presentasjon.pptx",
"krav-rikt-tekstformat.rtf",
"krav-tekstdokument.odt",
)
@dataclass
class StubGuard:
"""The pinned guard's surface, offline. A TEST DOUBLE -- never in `src/`."""
def __call__(self, text: str) -> GateDecision:
return GateDecision(sanitized_text=text, disposition=WARN, reasons=())
def test_the_drop_directory_holds_exactly_the_three_containers() -> None:
"""N = 3, and N is read off the directory rather than asserted from memory.
The generator and the fasit live one level UP on purpose: Door B walks this
directory recursively, so anything else parked here would enter the run and
the denominator would stop being three.
"""
assert sorted(path.name for path in DROP.iterdir()) == list(EXPECTED_FILES)
def test_all_three_go_through_door_b(tmp_path: Path) -> None:
"""3/3 merged, offline, through the same call the operator makes."""
result = process_inbox(
DROP,
tmp_path / "bundle",
INGESTED_AT,
okf_type="note",
gate=StubGuard(), # type: ignore[arg-type]
)
assert [item.source_file for item in result.persisted] == list(EXPECTED_FILES)
assert (result.quarantined, result.rejected, result.failed) == ((), (), ())
def test_every_persisted_concept_carries_the_documents_own_strings(tmp_path: Path) -> None:
"""A concept that arrived empty would still count as persisted.
So the count is not the whole assertion: each written concept has to carry
the title, a label from the pairing table and a cell from the grid -- one
string from each of the three structures the fixture is built out of.
"""
bundle = tmp_path / "bundle"
result = process_inbox(
DROP,
bundle,
INGESTED_AT,
okf_type="note",
gate=StubGuard(), # type: ignore[arg-type]
)
for item in result.persisted:
text = (bundle / item.path).read_text(encoding="utf-8")
assert "Kravspesifikasjon for tunnelbelysning" in text
assert "Årsdøgntrafikk:" in text
assert "Klasse C" in text
def test_the_fasit_is_hand_counted_and_committed_beside_the_fixtures() -> None:
"""The denominator the report divides by, frozen before the measurement ran."""
fasit = json.loads(FASIT.read_text(encoding="utf-8"))
assert sorted(fasit["documents"]) == list(EXPECTED_FILES)
for name in EXPECTED_FILES:
entry = fasit["documents"][name]
assert entry["cells"] == fasit["shared"]["cells"]
assert entry["pairs"] == fasit["shared"]["pairs"]
def test_three_synthetic_documents_do_not_make_a_row_measured() -> None:
"""One house style, N = 3, written by us -- that is exercise, not coverage."""
assert [_EVIDENCE[key] for key in (".pptx", ".odt", ".rtf")] == ["unmeasured"] * 3