open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity
llm-ingestion-okf/tests
History
Kjell Tore Guttormsen df5a1183c9
 feat(mcp): serve OKF bundles over MCP in two shapes, plus the generic skill 
The eval was written RED at `5f1772e` with no server in the tree. This is the
capability it was written against.

`okf mcp --bundle <dir>` serves exactly one bundle, whose tools take no bundle
argument. `okf mcp --root <dir>` (repeatable) serves every bundle under the
roots and knows NONE of them by name. Four tools -- `okf_list`,
`okf_describe`, `okf_ask`, `okf_fetch` -- each carrying its reason in the
description a client actually reads.

Gate today: 1 (7/7) - 2 (83/181) - 3 (4/4) - 4 (9/9) - 5 (3/3) - 6 (6/6),
`GATE RED: rows 2`, exit 1.

THE PROTOCOL IS STDLIB, AND THAT IS THE PACKAGING INVARIANT KEPT RATHER THAN
A TASTE. An MCP SDK would be this package's second runtime dependency on the
DEFAULT install path, for four JSON-RPC methods and a newline framing, and
`test_the_only_runtime_dependency_is_the_security_boundary` pins that list
literally. Chosen hand-written because the surface needed is `initialize`,
`notifications/initialized`, `tools/list` and `tools/call`; `uv.lock` is
untouched.

NOTHING IS CACHED ACROSS CALLS, and row 3 is why. Every call re-walks the
roots and recomputes `bundle_ref`, so a bundle added, removed or rebuilt while
the process runs is seen by the next call with no restart, no configuration
edit and no code change -- 9 of 9 discovery checks over three bundles written
while the server was serving. The cost is paid per call and is published
rather than hidden: 0.75 s for the identity of a 2 756-concept bundle, 5.6 s
for one ask, 4 min 13 s for row 2's full run over four bundles.

CONTAINMENT IS TWO INDEPENDENT CHECKS: the bundle's own index must name the
concept, AND `connectors.safe_resolve` must place it inside the bundle. A
mutant removing either one alone still refuses -- with a DIFFERENT code, which
row 6 asserts by name -- and one removing both is killed. Row 6 declares a
code set per case because its first run had the 10 MB concept refused as
`concept_unknown`: the fixture had not named the file in the index, so the
size ceiling never ran and the row was green for a reason unrelated to the
attack.

`okf card <bundle>` and `okf skill --generic` are the one-to-many skill
candidate. The card is DERIVED on every run and never written into the bundle:
storing it would move the bytes of all six `examples/*/expected-bundle` trees
(23 files compared byte-for-byte) and of the pinned reference bundle, to keep
something recomputable in under a second, and a stored card is one more
artefact that can disagree with what is beside it. Measured here rather than
taken from the order: two per-bundle skills are identical on 280 of 312 and
310 lines; the 62 that differ are identity, concept count, the
conditional-field table, the whole-bundle cost and the breaking point. The
generic skill carries none of them, and `render_generic()` takes no argument,
so there is no bundle it could have read.

Row 2 decomposes into three numbers and the middle one is the finding: 99 of
181 (bundle, anchor) pairs are present in the bundles at all, 83 of those 99
were reached, and 0 of 83 were met by `okf_fetch` on the anchor as a concept
id. The set's anchors and this library's concept ids are different
vocabularies, so every pair met was met through the ranker -- 83 is a FLOOR on
the ceiling, never the ceiling.

13 mutants in a scratch copy, never in the working tree: 12 killed, 1 survived
with its mechanism printed, 0 errors, control green first. Suite 2323 passed,
2 skipped. The architecture choice between the two shapes is the OPERATOR's;
these rows are its input. Report: docs/2026-09-20-mcp-to-varianter.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 10:25:55 +02:00
..
fixtures fix(retrieval-gate): rows 2 and 3 count against the pinned set, not the run 2026-09-19 21:03:55 +02:00
test_accounting_gate.py test(accounting-gate): the published door contract is behind the gate, and a doorless bundle is silent 2026-09-19 22:51:54 +02:00
test_adjudicate.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_asset_bundle.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_asset_gate.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_asset_import.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_asset_limits.py fix(assets): budget every link by what its decoder COSTS (0.10.1) 2026-09-18 19:00:38 +02:00
test_asset_structure.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
test_asset_viewable.py test(assets): the RLE8 rule is short of the independent decoder, in 703 of 22 620 streams 2026-09-19 22:46:13 +02:00
test_assets.py test(mutants,assets): a mutant is judged by the suite that owns it, 45 of 45 2026-09-19 19:33:23 +02:00
test_attested_computation.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_block_sources_flat_readers.py fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped 2026-09-12 16:42:22 +02:00
test_bold_title.py feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word 2026-09-09 23:05:45 +02:00
test_bundle_identity.py feat(consume): parent reaches the reader -- excerpt field, body link, checker rule 2026-09-11 12:36:23 +02:00
test_cid_measure.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_cli_build.py feat(cli,consume): the first span, and the fusion that punished fine-graining 2026-09-09 15:40:01 +02:00
test_cli_gate.py feat(accounting): okf build accounts for every source element 2026-09-17 18:35:20 +02:00
test_code_fence.py fix(gate,propose): okf build runs a real guard; a code fence declares no structure 2026-09-16 00:19:50 +02:00
test_content_accounting.py fix(accounting): a document refused whole is never clean, and the run says how many (H1) 2026-09-19 05:48:57 +02:00
test_contents_name_part.py feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
test_contract_check.py feat(check): a skill and a payload naming different bundles is a finding 2026-09-10 23:42:06 +02:00
test_converter_attribute_titles.py feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
test_corpus_run.py fix(extract,build): write a spreadsheet as pipe tables, stop linking the run log from the index 2026-09-08 10:06:58 +02:00
test_default_bundle_pin.py feat(consume): one source document took 8 of 8 delivered places, so cap it 2026-09-10 00:29:02 +02:00
test_depth_seven.py test(extract): a section below markdown's sixth level keeps its own level in the plan 2026-09-11 14:02:49 +02:00
test_docs_promises.py feat(assets): every carried image is one a model can be shown 2026-09-19 08:07:04 +02:00
test_document_identity.py feat(identity): an STS document's doc-number names its directory and its title the address 2026-09-11 03:08:30 +02:00
test_document_prior.py feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment 2026-09-09 14:17:00 +02:00
test_error_codes.py fix(assets): bound every link of the filter chain, and cover the backstop 2026-09-18 15:34:07 +02:00
test_exception_chaining.py test(errors): pin __cause__ preservation on every fail-fast wrap 2026-07-23 07:02:47 +02:00
test_extract.py test(extract): the file that IS the product had no reader, so state what one owes 2026-09-10 03:30:48 +02:00
test_faceted_index.py feat(profiles): a faceted index policy and the additive STRUCTURED_V1 profile 2026-08-27 00:30:42 +02:00
test_fidelity.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_file_connector.py feat(connectors): add the file connector with fail-closed path boundary 2026-07-16 19:55:08 +02:00
test_first_span.py feat(cli,consume): the first span, and the fusion that punished fine-graining 2026-09-09 15:40:01 +02:00
test_frontmatter_nesting.py fix(frontmatter): a nested key must not substitute for a top-level one 2026-08-31 23:31:53 +02:00
test_golden.py feat(okf-v0.2): D5 — the v0.2 golden fixture, with okf_version in root frontmatter 2026-07-31 17:27:48 +02:00
test_guard_adapter.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_http_connector.py feat(connectors): add the http connector and wire the network gate 2026-07-16 20:06:21 +02:00
test_import_consumable.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_import_facets.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_import_flow.py feat(import): Door C flow against an injected import gate (Phase 2 step 5) 2026-07-25 06:57:25 +02:00
test_inbox.py feat(inbox): fail-fast on filenames over the 255-byte limit 2026-07-25 06:27:43 +02:00
test_inbox_flow.py feat(inbox): walk the drop directory recursively 2026-09-07 04:11:00 +02:00
test_inbox_recursion.py feat(inbox): walk the drop directory recursively 2026-09-07 04:11:00 +02:00
test_inbox_structure.py feat(inbox): Door B derives structure and reprojects the index additively 2026-08-27 00:37:12 +02:00
test_index.py feat(index): maintain existing indexes on re-materialization (spec §6) 2026-07-16 20:04:21 +02:00
test_index_policy.py fix(profiles): P1-F1 — a permitted root key is not a required one 2026-07-31 18:16:13 +02:00
test_index_sort.py test(index): pin Door C's cross-run ordering bound 2026-09-01 19:58:19 +02:00
test_index_star_row.py feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment 2026-09-09 14:17:00 +02:00
test_k2_office_fixtures.py feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
test_link_line_signal.py feat(consume): the door's link line is out of the default reading 2026-09-12 18:56:53 +02:00
test_load_bearing.py feat(profiles): DEFAULT stamps commons' O2 generated, V1 executed 2026-08-09 12:29:05 +02:00
test_manifest.py feat(manifest): reject [/] in extraction title (ingest-spec §4) 2026-07-23 21:33:47 +02:00
test_materialize.py feat(profiles): DEFAULT stamps commons' O2 generated, V1 executed 2026-08-09 12:29:05 +02:00
test_mcp_gate.py feat(mcp): serve OKF bundles over MCP in two shapes, plus the generic skill 2026-09-20 10:25:55 +02:00
test_mcp_server.py feat(mcp): serve OKF bundles over MCP in two shapes, plus the generic skill 2026-09-20 10:25:55 +02:00
test_md_declared_headings.py feat(propose,cli): the coverage gap had one cause, and round 7's own decomposition did not reproduce 2026-09-09 17:43:56 +02:00
test_multi_source_provenance.py fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped 2026-09-12 16:42:22 +02:00
test_okf_consume.py feat(consume): the payload says what of the question it reached, and row 4 reads it 2026-09-20 08:44:06 +02:00
test_okf_consume_shim.py feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word 2026-09-09 23:05:45 +02:00
test_okf_project.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_okf_skill.py feat(check): a skill and a payload naming different bundles is a finding 2026-09-10 23:42:06 +02:00
test_okf_v0_2_characterization.py fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped 2026-09-12 16:42:22 +02:00
test_okf_v0_2_profile.py feat(profiles): DEFAULT stamps commons' O2 generated, V1 executed 2026-08-09 12:29:05 +02:00
test_okf_watch.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_outline_gate.py feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment 2026-09-09 14:17:00 +02:00
test_outline_measure.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_packaging.py fix(assets): bound what the run pays, not what the document claims (0.10.1) 2026-09-18 13:41:18 +02:00
test_pandoc_binary.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_parent_index.py test(structure): the index resolves a parent naming a segment of its own document 2026-09-11 13:48:11 +02:00
test_parent_reaches_reader.py feat(consume): parent reaches the reader -- excerpt field, body link, checker rule 2026-09-11 12:36:23 +02:00
test_parent_text.py docs(consume,build): both parent defaults stay off, and the reason is now a measurement 2026-09-11 13:39:50 +02:00
test_pdf_font_and_ocr.py feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off 2026-09-08 23:10:47 +02:00
test_pdf_heading_reserve.py feat(propose,cli): typography as a reserve, and the two of our own numbers it took to measure it 2026-09-09 00:25:51 +02:00
test_pdf_outline.py test(pdf): two bookmarks on one line, and nothing counts the one that is lost 2026-09-10 03:16:14 +02:00
test_profile.py feat(inbox): point every concept at the document it came from, with a locator per format 2026-09-08 14:39:24 +02:00
test_profile_threading.py test(profiles): pin the re-run promise V1 was published under 2026-08-09 12:39:23 +02:00
test_propose_segments.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_propose_sheet_sections.py feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
test_provenance.py fix(extract): HTML collapsed to one line, so 828 of 828 sections had no boundary 2026-09-09 23:57:47 +02:00
test_quality.py feat(quality): okf quality --fasit, boundary recall against a declared structure 2026-09-13 07:27:33 +02:00
test_render.py feat(render): add §5 body renderers as pure functions 2026-07-16 19:52:21 +02:00
test_render_hook.py feat(profiles): a profile may name a per-suffix renderer 2026-09-02 14:19:18 +02:00
test_retrieval_gate.py test(retrieval-gate): the pin needs the bytes, the threshold needs a range, and an empty hold-out clears nothing 2026-09-20 08:45:27 +02:00
test_root_frontmatter_emission.py feat(okf-v0.2): D5 — the v0.2 golden fixture, with okf_version in root frontmatter 2026-07-31 17:27:48 +02:00
test_round7_defaults.py feat(cli,consume): the first span, and the fusion that punished fine-graining 2026-09-09 15:40:01 +02:00
test_run_frontmatter.py fix(frontmatter): write a value a YAML reader reads back, and read both forms 2026-09-11 11:09:49 +02:00
test_segmentation.py feat(segmentation): parse the adjudication state a plan already carries 2026-09-02 14:44:06 +02:00
test_segmented_collisions.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_segmented_golden.py test(golden): a clean extract is not a checkout, and says so 2026-09-19 19:19:34 +02:00
test_segmented_identity.py fix(segmentation): hash the extracted text and let the plan key fire 2026-09-02 14:38:20 +02:00
test_segmented_inbox.py feat(inbox): accept one segmentation plan per document 2026-09-02 14:51:53 +02:00
test_segmented_index.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_segmented_okf_v0_2_golden.py test(examples): byte-pinned golden for SEGMENTED_OKF_V0_2 2026-09-02 14:24:05 +02:00
test_segmented_profile.py feat(profiles): SEGMENTED_OKF_V0_2 so a segmented bundle can declare its spec 2026-09-02 14:21:15 +02:00
test_segmented_rebuild.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_segmented_rounds.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_shared_id_prefix.py test(consume): a directory every concept shares must not order them 2026-09-11 04:43:37 +02:00
test_shell_parent.py fix(structure): the index resolves a parent naming a segment of its own document 2026-09-11 13:55:03 +02:00
test_soft_hyphen_door.py test(accounting): the log must say where the soft-hyphen count comes from (red) 2026-09-19 08:40:51 +02:00
test_span_gaps.py feat(propose,cli): the coverage gap had one cause, and round 7's own decomposition did not reproduce 2026-09-09 17:43:56 +02:00
test_sql_connector.py feat(connectors): add the sql connector (read-only sqlite, env-resolved) 2026-07-16 19:56:41 +02:00
test_stem_prefix.py feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word 2026-09-09 23:05:45 +02:00
test_strict_v1.py feat(phase-3): the index policy becomes configurable, with the reader that judges it 2026-07-25 20:30:54 +02:00
test_structure.py fix(structure): a derived reference needs a cue, not just a number shape 2026-08-29 09:26:57 +02:00
test_sts_description.py test(description): a spec point a YAML reader cannot read verbatim is not written 2026-09-11 03:46:11 +02:00
test_table_measure.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_tie_shared_rank.py chore(ruff): the acceptance was whatever the default happened to be [skip-docs] 2026-09-09 23:15:17 +02:00
test_title_covered.py test(consume): the lookup partition still wins, and the CLI defaults agree with the signature 2026-09-10 22:20:42 +02:00
test_title_covered_rise.py test(consume): a short title the question happens to cover passes the section it names 2026-09-11 01:23:14 +02:00
test_xml_declared_route.py test(propose): a declared XML structure does not take the route declared structure takes 2026-09-10 06:55:00 +02:00
test_xml_sts.py test(extract): an exact invariant does not get to be 0.999998 2026-09-10 04:07:35 +02:00
test_yaml_frontmatter.py fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped 2026-09-12 16:42:22 +02:00