Kjell Tore Guttormsen
446d97656a
Order 20260904T172353Z-6290714297-from-.claude. 39/43 K2/trinn1 documents measured with tools/okf_cid_measure.py; the 4 not measured are the same 4 K1 already named as coded rejections. Over 10 %: 1/39. Over 50 %: 1/39 -- Bilag 9.1 both times, at 95.0743 % (reproduces the published 95.1 %). No other document in the corpus crosses either line. Known-negative control holds (a docx gives 0 CID characters on a nonzero total). The word count (44) does not reproduce the earlier ad hoc 98 -- no script for that figure was ever committed, so nothing exists to reconcile against; both counts are gibberish letter runs, not prose, so the substantive finding does not move. K2 bundle hashed before and after, byte-unchanged (1108 files). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
8.6 KiB
CID-glyph share across K2 (trinn 1, N = 43), 2026-09-04
Order 20260904T172353Z-6290714297-from-.claude. Measure, don't build: no
CID-mapped extraction path, no new extractor, no change to the K2 bundle.
The question
docs/2026-09-04-k3-arm-c.md found, while measuring something else, that
Bilag 9.1 -- the largest concept in the K2 bundle and the one the consumer
flagged -- is an extraction failure: 95.1 % of its extracted text is
(cid:N) glyph codes, the placeholder pdfminer.six (behind pdfplumber,
this library's PDF reader) emits when a font carries no usable ToUnicode
mapping. The operator's question before anything is built: is Bilag 9.1
alone, or does the CID failure reach other documents -- a K1-arm
(door-level) question, not a K3 one?
Method
tools/okf_cid_measure.py, added with a red-first test
(tests/test_cid_measure.py) pinning measure() against fixture text of
known composition before the implementation existed. It calls
llm_ingestion_okf.extract.extract_text -- the exact call the door makes --
never a second parser, so the number answers "what does this repository
actually persist," not something a different code path would produce.
.venv/bin/python tools/okf_cid_measure.py \
--corpus ~/corpora/okf-telling-20260829/K2/trinn1 \
--report /tmp/cid-k2.md
N = 43 is the corpus directory's file count, verified against ground truth
with ls ~/corpora/okf-telling-20260829/K2/trinn1 | wc -l -> 43, the same
denominator K1 used (docs/2026-09-02-korpuskjoring-k1-k2.md).
Two definitions are this instrument's own, stated rather than assumed:
a CID glyph code is the literal pattern (cid:\d+); an alphabetic word
of four or more letters is a maximal run of Unicode letters of that length
(so æøå count, digits and _ don't, and (cid:12)'s own "cid" -- three
letters -- can never qualify).
Controls (Verifiseringsloven ansikt 4 -- absence is a measurement, not a fact)
- Known-positive, Bilag 9.1 reproduces 95.1 %. Measured here: 206 758 of 217 470 characters are CID codes = 95.0743 %, which rounds to the published figure. Reproduced exactly on the metric that has a stated definition.
- Known-positive word count does NOT reproduce, and the reason is stated.
This instrument counts 44 words of 4+ letters on Bilag 9.1, not the
previously reported 98. No script or command behind the 98 figure was ever
committed -- the same gap this repository's own
tools/okf_fidelity.pycriticizes in the 2026-08-29 Arm A report's uncommitted docx/xlsx figures (docx 193/196,xlsx 193/193, "those figures could not be re-measured... against the product path, or at all"). There is nothing to reconcile against, so the discrepancy is reported rather than resolved. It changes nothing substantive: this instrument's own 44 "words" are gibberish letter runs (gjbj,jklkmnm,qrsttuvwx,wxrzrx), not prose, under either count -- the document is unreadable either way. - Known-negative: at least one docx gives 0 CID characters with a nonzero
total.
Del I Vedlegg 1 Søknadsbrev.docx: 15 506 total characters, 0 CID characters. Confirmed; every measured non-PDF file in the corpus gives 0.
The denominator
39/43 measured. The 4 not measured are the same 4 K1 already named as coded rejections, for the same reasons:
| file | reason |
|---|---|
Del II Bilag 2.10 - Samlemodell-SSU.smc |
extractor_unknown -- .smc has no registered extractor |
Del II Bilag 2.11 - Eksisterende bygg dwg-tegninger.zip |
extractor_unknown -- .zip has no registered extractor |
Del II Bilag 8 Mal for tiltransportavtale.doc |
extractor_unknown -- legacy .doc has no registered extractor (only .docx) |
Del II Bilag 2.4 - VA-tegninger.pdf |
extractor_empty_pdf -- no text on any page |
A file this instrument cannot measure contributes no CID characters and no total characters either -- it is absent from the sums below, not counted as 0 %.
The two sums
Over 10 %: 1/39. Over 50 %: 1/39. The same one document both times: Bilag 9.1. No other document in the 39 measured crosses either line; the next-highest CID share in the corpus is 0.0 %.
Full table
| file | total chars | cid chars | cid share | words (4+ letters) |
|---|---|---|---|---|
| Del I Konkurransebeskrivelsen - konkurranse med forhandling.pdf | 42781 | 0 | 0.0 % | 2729 |
| Del I Vedlegg 1 Søknadsbrev.docx | 15506 | 0 | 0.0 % | 188 |
| Del I Vedlegg 2 Forpliktelseserklæring ved bruk av underleverandører.docx | 2005 | 0 | 0.0 % | 82 |
| Del I Vedlegg 3 Erfaringsoversikt.docx | 5370 | 0 | 0.0 % | 42 |
| Del I Vedlegg 4 Tilbudsbrev.docx | 8223 | 0 | 0.0 % | 413 |
| Del I Vedlegg 5 Begrunnelse for å bruke krav i stedet for tildelingskriterier.pdf | 22746 | 0 | 0.0 % | 1809 |
| Del II Kontraktsgrunnlaget.pdf | 73028 | 0 | 0.0 % | 5579 |
| Del II Bilag 0 Dokumentliste del II.xlsx | 3031 | 0 | 0.0 % | 117 |
| Del II Bilag 1 - Kravspesifikasjon med funksjonsbeskrivelse alle fag.pdf | 251069 | 0 | 0.0 % | 21558 |
| Del II Bilag 1.1 - Stange skole - Generelle tekniske krav.pdf | 31000 | 0 | 0.0 % | 2644 |
| Del II Bilag 1.2 - Renholdstekniske funksjonskrav Stange kommune.pdf | 26873 | 0 | 0.0 % | 2326 |
| Del II Bilag 1.3 - BIM-manual.pdf | 28935 | 0 | 0.0 % | 1968 |
| Del II Bilag 1.4 - Teknisk sjekkliste – A20 Miljøgiftslisten.pdf | 5795 | 0 | 0.0 % | 483 |
| Del II Bilag 2.1 - Arkitekttegninger.pdf | 158951 | 0 | 0.0 % | 9424 |
| Del II Bilag 2.10 - Samlemodell-SSU.smc | -- | -- | -- | not measured: extractor_unknown |
| Del II Bilag 2.11 - Eksisterende bygg dwg-tegninger.zip | -- | -- | -- | not measured: extractor_unknown |
| Del II Bilag 2.2 - Landskapsplan L01.pdf | 4218 | 0 | 0.0 % | 296 |
| Del II Bilag 2.3 - Riggplan.pdf | 2185 | 0 | 0.0 % | 130 |
| Del II Bilag 2.4 - VA-tegninger.pdf | -- | -- | -- | not measured: extractor_empty_pdf |
| Del II Bilag 2.5 - Konstruksjonstegninger.pdf | 11214 | 0 | 0.0 % | 570 |
| Del II Bilag 2.6 - VVS-tegninger.pdf | 27059 | 0 | 0.0 % | 1169 |
| Del II Bilag 2.7 - Eksisterende EL- infrastruktur i grunn.pdf | 4038 | 0 | 0.0 % | 357 |
| Del II Bilag 2.8 - Branntegninger.pdf | 10275 | 0 | 0.0 % | 652 |
| Del II Bilag 2.9 - Lås og sikring.pdf | 15561 | 0 | 0.0 % | 208 |
| Del II Bilag 3.1 - Miljøteknisk rapport med tiltaksplan.pdf | 267546 | 0 | 0.0 % | 14550 |
| Del II Bilag 3.2.1 - RIG-R01 Datarapport.pdf | 71255 | 0 | 0.0 % | 5536 |
| Del II Bilag 3.2.2 - RIG-R02 Geoteknisk og ingeniørgeologisk fagrapport.pdf | 55454 | 0 | 0.0 % | 3887 |
| Del II Bilag 3.3.1 - Brannkonsept.pdf | 44177 | 0 | 0.0 % | 3469 |
| Del II Bilag 3.3.2 - Brannalarmorganisering.pdf | 23488 | 0 | 0.0 % | 2046 |
| Del II Bilag 3.4 - Premissdokument bygningsfysikk.pdf | 42894 | 0 | 0.0 % | 3601 |
| Del II Bilag 3.5 - Dokumentasjon av energibehov.pdf | 17363 | 0 | 0.0 % | 1213 |
| Del II Bilag 3.6 - Premissrapport akustikk.pdf | 57714 | 0 | 0.0 % | 3220 |
| Del II Bilag 3.7 - Miljøkartlegging Administrasjonsbygg.pdf | 95595 | 0 | 0.0 % | 6322 |
| Del II Bilag 3.8 - Overvannsutredning.pdf | 13761 | 0 | 0.0 % | 1093 |
| Del II Bilag 3.9 - Kravspesfikasjon solcelleanlegg.pdf | 21233 | 0 | 0.0 % | 1819 |
| Del II Bilag 4 - Stange skole Romliste teknisk.pdf | 6390 | 0 | 0.0 % | 330 |
| Del II Bilag 5 - SHA-Plan Stange skole utvidelse.pdf | 14076 | 0 | 0.0 % | 1174 |
| Del II Bilag 6 - Teknisk oppsett.pdf | 8517 | 0 | 0.0 % | 592 |
| Del II Bilag 7 Prisskjema.xlsx | 100694 | 0 | 0.0 % | 657 |
| Del II Bilag 8 Mal for tiltransportavtale.doc | -- | -- | -- | not measured: extractor_unknown |
| Del II Bilag 9.1 - Avtale som tiltransporteres Norconsult AS.pdf | 217470 | 206758 | 95.1 % | 44 |
| Del II Bilag 9.2 - Avtale som tiltransporteres ASAS Arkitektur AS.pdf | 20607 | 0 | 0.0 % | 1619 |
| Dokument for avtaleinngåelse.docx | 2170 | 0 | 0.0 % | 77 |
Verdict
Bilag 9.1 is alone. No other document in the 39 measured crosses 10 %, let alone 50 %. This is not a K1-arm (door-level) problem -- it is one document's fonts, in one corpus, with no usable ToUnicode mapping. Recommendation to the operator: park it. A CID-mapped extraction path bought for one document out of 43 is scope this order correctly did not ask to build.
The K2 bundle is unchanged
This work reads the source corpus only; it never opens the bundle. Hashed
before and after anyway, same method as docs/2026-09-04-k3-arm-c.md, whole
tree, 1108 files (first attempt without LC_ALL=C produced a different
aggregate hash from locale-dependent sort order alone -- per-file bytes were
never in question; pinning the locale reproduces the recorded hash exactly):
LC_ALL=C find K2-bundle-20260903 -type f | LC_ALL=C sort | xargs shasum -a 256 | shasum -a 256
9cd745194346cda0c70eab9c7136fa44506203bbe85bc17d7eff2766c6e9b4d1 (before and after)