open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity
llm-ingestion-okf/tests
History
Kjell Tore Guttormsen 2c8296b807
 test(consume): the retrieval gate, written red over nine rows 
One command, one exit code, and no capability: nothing here changes the
ranking, the fusion, the tokenisation or the cut. For a frozen question set
against a bundle it asks how many of N measurement units the payload carries
the fasit for, gives every miss exactly ONE class, and asks the thing the
payload does not do at all -- say so when it does not know.

Today, on the synthetic corpus this repository generates and the five sets it
ships pinned by sha256: rows 1 (9 of 9), 2 (7 of 7) and 6 (9 of 9) green;
3 (2 of 5), 4 (3 of 6), 5 (0 of 1), 7 (11 of 13), 8 (not run) and 9 (0 of 6)
red. GATE RED, exit 1.

- Row 3 is the finding to read first. In a bundle built from ONE source
  document, every concept past the first two carries that document's
  `source_file`, so a concept the RANK had already lost is withheld as
  `source_quota_exceeded`. The truth is decided by the SAME cut run without
  the quota -- PM measured the same lie on 13 of 25 real misses 2026-09-17.
- Row 4 is the second. No payload key says "this bundle does not answer
  that", so the only honest reading today is `delivered == 0`; three of six
  controls come out right and the three that do not get eight excerpts and no
  statement. A control the bundle DOES answer must come back unmarked, or the
  marking says nothing, so both directions sit in the same row.
- A miss has exactly one class -- (a) not in the bundle, (b) below k, (c) cut
  by quota or budget, (d) delivered without the citation, (e) other -- and
  each is FORCED by its own fixture. An unplaced miss takes the row to 0 of N.
- The judge opens the bundle. Every concept is read off disk before a payload
  is built, a delivered excerpt whose text is not the concept file's bytes is
  not a hit, and the two title forms come from `quality` rather than being
  written a second time.
- FELLED means a row got WORSE, never merely that something changed: removing
  the quota makes row 3 green, and counting that as a kill would credit this
  gate with a check it does not have. The two survivors are reported with what
  they measured -- the document prior dead and RRF_K flattened each moved 0
  ranks and 0 deliveries on these fixtures -- and with a mechanism for each.
- Rows 8 and 9 are never green by leaving something out. The three real sets
  are read from path + sha256 and never committed: a gold set names a
  consumer's documents and this repository is public. K2 is 0 of 6 because the
  gold set exists nowhere while its bundles do.

Denominators re-measured against the sources 2026-09-19, and all four hold:
wiki 20 questions / 29 fasit entries (sha 972d0f57...), R761 8 (S1-S6 + KP +
KN), vegnormal 32 questions / 43 must_cite, K2 6. PM's 45 of 70 is itself a
mixed sum -- wiki at question level, vegnormal at citation level -- so this
gate carries it labelled as recorded and never sums the two granularities.

46 tests, each failing row driven both ways through inputs rather than by
editing the row. No network (the socket guard is fired before its silence
counts), no private corpus, no clock.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-19 10:47:21 +02:00
..
fixtures test(consume): the retrieval gate, written red over nine rows 2026-09-19 10:47:21 +02:00
test_accounting_gate.py test(accounting): the two forged routes become mutants, and src/ becomes measurable 2026-09-19 09:47:46 +02:00
test_adjudicate.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_asset_bundle.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_asset_gate.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_asset_import.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_asset_limits.py fix(assets): budget every link by what its decoder COSTS (0.10.1) 2026-09-18 19:00:38 +02:00
test_asset_structure.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_asset_viewable.py refactor(assets,accounting): drop two checks no mutant can fell, add the delta cut 2026-09-19 09:36:18 +02:00
test_assets.py feat(assets): every carried image is one a model can be shown 2026-09-19 08:07:04 +02:00
test_attested_computation.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_block_sources_flat_readers.py fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped 2026-09-12 16:42:22 +02:00
test_bold_title.py feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word 2026-09-09 23:05:45 +02:00
test_bundle_identity.py feat(consume): parent reaches the reader -- excerpt field, body link, checker rule 2026-09-11 12:36:23 +02:00
test_cid_measure.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_cli_build.py feat(cli,consume): the first span, and the fusion that punished fine-graining 2026-09-09 15:40:01 +02:00
test_cli_gate.py feat(accounting): okf build accounts for every source element 2026-09-17 18:35:20 +02:00
test_code_fence.py fix(gate,propose): okf build runs a real guard; a code fence declares no structure 2026-09-16 00:19:50 +02:00
test_content_accounting.py fix(accounting): a document refused whole is never clean, and the run says how many (H1) 2026-09-19 05:48:57 +02:00
test_contents_name_part.py feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
test_contract_check.py feat(check): a skill and a payload naming different bundles is a finding 2026-09-10 23:42:06 +02:00
test_converter_attribute_titles.py feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
test_corpus_run.py fix(extract,build): write a spreadsheet as pipe tables, stop linking the run log from the index 2026-09-08 10:06:58 +02:00
test_default_bundle_pin.py feat(consume): one source document took 8 of 8 delivered places, so cap it 2026-09-10 00:29:02 +02:00
test_depth_seven.py test(extract): a section below markdown's sixth level keeps its own level in the plan 2026-09-11 14:02:49 +02:00
test_docs_promises.py feat(assets): every carried image is one a model can be shown 2026-09-19 08:07:04 +02:00
test_document_identity.py feat(identity): an STS document's doc-number names its directory and its title the address 2026-09-11 03:08:30 +02:00
test_document_prior.py feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment 2026-09-09 14:17:00 +02:00
test_error_codes.py fix(assets): bound every link of the filter chain, and cover the backstop 2026-09-18 15:34:07 +02:00
test_exception_chaining.py test(errors): pin __cause__ preservation on every fail-fast wrap 2026-07-23 07:02:47 +02:00
test_extract.py test(extract): the file that IS the product had no reader, so state what one owes 2026-09-10 03:30:48 +02:00
test_faceted_index.py feat(profiles): a faceted index policy and the additive STRUCTURED_V1 profile 2026-08-27 00:30:42 +02:00
test_fidelity.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_file_connector.py feat(connectors): add the file connector with fail-closed path boundary 2026-07-16 19:55:08 +02:00
test_first_span.py feat(cli,consume): the first span, and the fusion that punished fine-graining 2026-09-09 15:40:01 +02:00
test_frontmatter_nesting.py fix(frontmatter): a nested key must not substitute for a top-level one 2026-08-31 23:31:53 +02:00
test_golden.py feat(okf-v0.2): D5 — the v0.2 golden fixture, with okf_version in root frontmatter 2026-07-31 17:27:48 +02:00
test_guard_adapter.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_http_connector.py feat(connectors): add the http connector and wire the network gate 2026-07-16 20:06:21 +02:00
test_import_consumable.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_import_facets.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_import_flow.py feat(import): Door C flow against an injected import gate (Phase 2 step 5) 2026-07-25 06:57:25 +02:00
test_inbox.py feat(inbox): fail-fast on filenames over the 255-byte limit 2026-07-25 06:27:43 +02:00
test_inbox_flow.py feat(inbox): walk the drop directory recursively 2026-09-07 04:11:00 +02:00
test_inbox_recursion.py feat(inbox): walk the drop directory recursively 2026-09-07 04:11:00 +02:00
test_inbox_structure.py feat(inbox): Door B derives structure and reprojects the index additively 2026-08-27 00:37:12 +02:00
test_index.py feat(index): maintain existing indexes on re-materialization (spec §6) 2026-07-16 20:04:21 +02:00
test_index_policy.py fix(profiles): P1-F1 — a permitted root key is not a required one 2026-07-31 18:16:13 +02:00
test_index_sort.py test(index): pin Door C's cross-run ordering bound 2026-09-01 19:58:19 +02:00
test_index_star_row.py feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment 2026-09-09 14:17:00 +02:00
test_k2_office_fixtures.py feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
test_link_line_signal.py feat(consume): the door's link line is out of the default reading 2026-09-12 18:56:53 +02:00
test_load_bearing.py feat(profiles): DEFAULT stamps commons' O2 generated, V1 executed 2026-08-09 12:29:05 +02:00
test_manifest.py feat(manifest): reject [/] in extraction title (ingest-spec §4) 2026-07-23 21:33:47 +02:00
test_materialize.py feat(profiles): DEFAULT stamps commons' O2 generated, V1 executed 2026-08-09 12:29:05 +02:00
test_md_declared_headings.py feat(propose,cli): the coverage gap had one cause, and round 7's own decomposition did not reproduce 2026-09-09 17:43:56 +02:00
test_multi_source_provenance.py fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped 2026-09-12 16:42:22 +02:00
test_okf_consume.py fix(skill): regenerate skills/okf-consume from the golden bundle it ships a payload for 2026-09-11 02:21:35 +02:00
test_okf_consume_shim.py feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word 2026-09-09 23:05:45 +02:00
test_okf_project.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_okf_skill.py feat(check): a skill and a payload naming different bundles is a finding 2026-09-10 23:42:06 +02:00
test_okf_v0_2_characterization.py fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped 2026-09-12 16:42:22 +02:00
test_okf_v0_2_profile.py feat(profiles): DEFAULT stamps commons' O2 generated, V1 executed 2026-08-09 12:29:05 +02:00
test_okf_watch.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_outline_gate.py feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment 2026-09-09 14:17:00 +02:00
test_outline_measure.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_packaging.py fix(assets): bound what the run pays, not what the document claims (0.10.1) 2026-09-18 13:41:18 +02:00
test_pandoc_binary.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_parent_index.py test(structure): the index resolves a parent naming a segment of its own document 2026-09-11 13:48:11 +02:00
test_parent_reaches_reader.py feat(consume): parent reaches the reader -- excerpt field, body link, checker rule 2026-09-11 12:36:23 +02:00
test_parent_text.py docs(consume,build): both parent defaults stay off, and the reason is now a measurement 2026-09-11 13:39:50 +02:00
test_pdf_font_and_ocr.py feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off 2026-09-08 23:10:47 +02:00
test_pdf_heading_reserve.py feat(propose,cli): typography as a reserve, and the two of our own numbers it took to measure it 2026-09-09 00:25:51 +02:00
test_pdf_outline.py test(pdf): two bookmarks on one line, and nothing counts the one that is lost 2026-09-10 03:16:14 +02:00
test_profile.py feat(inbox): point every concept at the document it came from, with a locator per format 2026-09-08 14:39:24 +02:00
test_profile_threading.py test(profiles): pin the re-run promise V1 was published under 2026-08-09 12:39:23 +02:00
test_propose_segments.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_propose_sheet_sections.py feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
test_provenance.py fix(extract): HTML collapsed to one line, so 828 of 828 sections had no boundary 2026-09-09 23:57:47 +02:00
test_quality.py feat(quality): okf quality --fasit, boundary recall against a declared structure 2026-09-13 07:27:33 +02:00
test_render.py feat(render): add §5 body renderers as pure functions 2026-07-16 19:52:21 +02:00
test_render_hook.py feat(profiles): a profile may name a per-suffix renderer 2026-09-02 14:19:18 +02:00
test_retrieval_gate.py test(consume): the retrieval gate, written red over nine rows 2026-09-19 10:47:21 +02:00
test_root_frontmatter_emission.py feat(okf-v0.2): D5 — the v0.2 golden fixture, with okf_version in root frontmatter 2026-07-31 17:27:48 +02:00
test_round7_defaults.py feat(cli,consume): the first span, and the fusion that punished fine-graining 2026-09-09 15:40:01 +02:00
test_run_frontmatter.py fix(frontmatter): write a value a YAML reader reads back, and read both forms 2026-09-11 11:09:49 +02:00
test_segmentation.py feat(segmentation): parse the adjudication state a plan already carries 2026-09-02 14:44:06 +02:00
test_segmented_collisions.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_segmented_golden.py feat(examples): SEGMENTED_V1 golden fixture and decision record 2026-09-01 00:31:15 +02:00
test_segmented_identity.py fix(segmentation): hash the extracted text and let the plan key fire 2026-09-02 14:38:20 +02:00
test_segmented_inbox.py feat(inbox): accept one segmentation plan per document 2026-09-02 14:51:53 +02:00
test_segmented_index.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_segmented_okf_v0_2_golden.py test(examples): byte-pinned golden for SEGMENTED_OKF_V0_2 2026-09-02 14:24:05 +02:00
test_segmented_profile.py feat(profiles): SEGMENTED_OKF_V0_2 so a segmented bundle can declare its spec 2026-09-02 14:21:15 +02:00
test_segmented_rebuild.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_segmented_rounds.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_shared_id_prefix.py test(consume): a directory every concept shares must not order them 2026-09-11 04:43:37 +02:00
test_shell_parent.py fix(structure): the index resolves a parent naming a segment of its own document 2026-09-11 13:55:03 +02:00
test_soft_hyphen_door.py test(accounting): the log must say where the soft-hyphen count comes from (red) 2026-09-19 08:40:51 +02:00
test_span_gaps.py feat(propose,cli): the coverage gap had one cause, and round 7's own decomposition did not reproduce 2026-09-09 17:43:56 +02:00
test_sql_connector.py feat(connectors): add the sql connector (read-only sqlite, env-resolved) 2026-07-16 19:56:41 +02:00
test_stem_prefix.py feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word 2026-09-09 23:05:45 +02:00
test_strict_v1.py feat(phase-3): the index policy becomes configurable, with the reader that judges it 2026-07-25 20:30:54 +02:00
test_structure.py fix(structure): a derived reference needs a cue, not just a number shape 2026-08-29 09:26:57 +02:00
test_sts_description.py test(description): a spec point a YAML reader cannot read verbatim is not written 2026-09-11 03:46:11 +02:00
test_table_measure.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_tie_shared_rank.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_title_covered.py test(consume): the lookup partition still wins, and the CLI defaults agree with the signature 2026-09-10 22:20:42 +02:00
test_title_covered_rise.py test(consume): a short title the question happens to cover passes the section it names 2026-09-11 01:23:14 +02:00
test_xml_declared_route.py test(propose): a declared XML structure does not take the route declared structure takes 2026-09-10 06:55:00 +02:00
test_xml_sts.py test(extract): an exact invariant does not get to be 0.999998 2026-09-10 04:07:35 +02:00
test_yaml_frontmatter.py fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped 2026-09-12 16:42:22 +02:00