open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(build): name the documents the gate refused whole, and close as 1.0.0

Browse source 
Measured 2026-09-20 on an official documentation corpus of 594 sources built
with the shipped default gate `guard-trusted-source`: 17 sources were refused
OUTRIGHT -- `fail_secure` 3, `quarantine_review` 14 -- and 16 of them were
among 197 official documentation pages, the pages on hooks, skills,
permissions, errors, env-vars and authentication among them. The summary said
only `fail_secure`: 3/594. Three of the four facts a reader needs were missing:
the COUNT of documents the gate dropped (the existing `rejected (coded)` line
sums gate refusals and extraction failures, two failures with two different
remedies), the NAMES, and the way out. Rebuilt with `--gate none`, all 17 went
through untouched, so the refusal is the gate and not the readers.

`okf build` now prints a `Documents the gate refused WHOLE` section directly
under the denominator, carrying all four: the count with its denominator, the
names capped at ten with the rest in the bundle's `log.md`, the codes, and
`--gate none` for a source you vouch for yourself. The same fact goes to stderr
in one line, built from the same field, because `okf build > report.txt` is an
ordinary thing to do. `log.md` gains one bullet naming every refused document,
uncapped.

The exit code deliberately does not move. The build is valid -- every refusal
is coded, the conservation identity holds, and the bundle is a true record of
what the gate allowed. What was wrong was the silence.

A run the gate refused nothing from is byte-identical in both places, which is
the known-negative in the new suite: no bundle this repository ships was built
with a gate refusal, so this cannot have moved a byte measured here.

Also, and measuring nothing new:

- README gains `Known limitations` high up -- the gate's refusals and the way
  out, the absent ceiling on what one run pays for images (a 70 KB PDF with 16
  images under the declared limit reached 851 MB peak RSS; RLIMIT_AS is not
  enforceable on this platform, so the 512 MiB per-link budget is the whole
  bound), the three gates of this repository that are RED today (retrieval
  5/7/8/9, MCP 2, accounting 2/3/6 -- all three re-run on this commit), what
  the content accounting does not count, and the rough edges nothing is
  planned for.
- The two `pip install` lines under "Install in detail" install `[extract]`.
  The first screen does; those two did not, so the two recipes produced
  different installations and the detailed one reports `resolved converter
  path: unresolved (extractor_extra_missing)`.
- Version `1.0.0`, synced across pyproject, `__version__`, `uv.lock`, the four
  README install lines, the install prose, the current-tag entry and the
  CHANGELOG, where the two "after the 0.10.1 notes were written, untagged"
  sections are folded in. It adds no capability over `v0.10.1`; what it adds
  is that the tool says what it does not do.

Suite: 2325 passed, 2 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-20 21:44:49 +02:00
commit e351e2d893
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
10 changed files with 512 additions and 69 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.gitignore vendored
View file

@ -18,5 +18,6 @@ build/
# Local-only: public remote — never publish session state or internal briefs
/STATE.md
/PLAN.md
/docs/oppstartsprompt.md
/.claude/projects/

153
CHANGELOG.md
View file

@ -5,6 +5,100 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [1.0.0] — 2026-09-20
### Added
- **A document the gate refuses WHOLE is named in the run's own summary.**
Measured 2026-09-20 on an official documentation corpus of 594 sources built
with the shipped default gate: 17 sources were refused outright, 16 of them
among 197 documentation pages, and the summary said only
`` `fail_secure`: 3/594 `` and `` `quarantine_review`: 14/594 ``. The count of
documents the gate dropped was not there (`rejected (coded)` sums gate
refusals and extraction failures, which have different remedies), the names
were not there, and neither was the way out. `okf build` now prints a
`Documents the gate refused WHOLE` section carrying all four — the count with
its denominator, the names (capped at ten, with the rest in the bundle's
`log.md`), the codes, and `--gate none` for a source you vouch for yourself —
and repeats it in one line on stderr, where a redirected stdout cannot hide
it. `log.md` gains a bullet naming every refused document, uncapped. **The
exit code does not move**: the build is valid, every refusal is coded and the
bundle is a true record of what the gate allowed; what was wrong was the
silence. A run the gate refused nothing from is byte-identical, in the
summary and in `log.md`.
- **A `Known limitations` section on the front page**, high up and before the
install detail: the gate's measured refusals and the way out, the absent
ceiling on what one run pays for images, the three gates of this repository
that are RED and what each red row means for a user, what the content
accounting does not count, and the rough edges nothing is planned for. No new
measurement — every number was already taken.
- **The payload says what of the question it reached** — a new top-level
`coverage` member carrying three lists: the terms the pre-pass read the
question as, the terms no concept in the bundle answers, and the terms no
delivered excerpt answers. Without it a reader holding eight excerpts cannot
tell a bundle that ANSWERED its question from one that merely ranked
something; the two payloads have the same shape. Documented as SS 8 point 7
of `docs/consumption-contract.md`, and the generated consumption skill is
told to read it.
- **Facts, and no verdict, which is a measurement rather than caution.** Two
readings were built and both falsified over **81 questions** (16 synthetic,
65 across three real gold sets, 2026-09-20): the share of a question's
terms a delivered excerpt answers separates the synthetic controls at 0.33
against 0.50 and then REVERSES on real data, where covered questions run
down to 0.27 while one genuinely uncovered question sits at 0.71; and the
share of a bundle tying the best lexical match is ~0.00 for every question
in a large bundle, covered or not. Question style dominates the first and
corpus size the second, so a pre-pass emitting a verdict would assert
across corpora what was measured on one.
- **Contract change, and the cost to a consumer is a re-run.** Every payload
grows the member; the checker does not read it, so a third-party pre-pass
that omits it stays conformant. The SS 7.4 known-positive moves with the
document it is measured on (14 721 / 375 → 16 389 / 417).
- **The retrieval gate is measurable where it was assertable**
(`tools/okf_retrieval_gate.py`, not shipped in the wheel):
- Row 8 prints the identity of every bundle it measured — path,
`bundle_id` and content ref — beside the set's sha256. Measured the same
day: two wiki bundles carrying the one `bundle_id` `claude-code-wiki` at
different refs score **6 of 20** and **5 of 20** on the same pinned set.
- `REAL_SET_PINS` states what each of the three real sets IS — questions,
fasit entries, controls and sha256 — so a self-written file in the right
shape is refused instead of reading `1 of 1 | 3 of 3 | GREEN`.
- Row 5 reads the hold-out threshold as a number in [0, 1] and RUNS the
registered set against the registered bundle, printing
`answered of asked = share against threshold`. `bool(threshold)` was the
whole check, so `report-only; any number is acceptable for v1` passed it.
- Row 4's marking reads the payload's `coverage`: `UNANSWERED_BAR = 2/3`,
swept and collapsing at both ends (at 0.50 eleven real covered questions
are marked; at 0.70 the row falls to 5 of 6). The margin is thin — 0.6087
against 0.6667 — and what it does not catch is published with it.
### Changed
- **The two `pip install` lines under "Install in detail" install
`[extract]`.** The first screen installs `llm-ingestion-okf[extract]` and
those two omitted it, so a reader following the detailed instructions got a
build that reports `resolved converter path: unresolved
(extractor_extra_missing)` and reads no binary format. Two recipes, two
different installations.
- **Version `1.0.0`.** The scope this tool is finished at. It adds no
capability over `v0.10.1`; what it adds is that the tool says what it does
not do. After this tag the library is touched for defects found in its own
use, and the next round is Google OKF v0.3.
- **A withheld concept now carries the rule that actually decided it.** The
source quota filters the WHOLE ranked candidate list rather than the top
`k`, so every over-quota candidate came back `source_quota_exceeded` —
including the ones the RANK had already put outside `k`, which the quota
only reached because it ran first. Measured on 25 real misses 2026-09-17:
**13 of them** were labelled by the quota and decided by the rank.
`consume._fates_without_quota` asks the same cut what would have become of
each candidate with no quota in force, and the drop keeps THAT rule; only a
candidate the quota-off cut would have delivered is named as the quota's.
The budget step is lifted into `consume._pack` and used by both, so the
quota-off fate is decided by the code the run itself uses. This moves the
`rule` string a consumer reads for some withheld entries; no delivery, no
rank and no excerpt byte moves, and no committed payload in this repository
changed. The retrieval gate's row 3 goes **2 of 5 RED to 5 of 5 GREEN**.
## [0.10.1] — 2026-09-19
### Removed
@ -626,65 +720,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
"degenerate" means zero characters after stripping whitespace and a pointer
block is text.
### Changed after the 0.10.1 notes were written (2026-09-20, untagged)
- **A withheld concept now carries the rule that actually decided it.** The
source quota filters the WHOLE ranked candidate list rather than the top
`k`, so every over-quota candidate came back `source_quota_exceeded` —
including the ones the RANK had already put outside `k`, which the quota
only reached because it ran first. Measured on 25 real misses 2026-09-17:
**13 of them** were labelled by the quota and decided by the rank.
`consume._fates_without_quota` asks the same cut what would have become of
each candidate with no quota in force, and the drop keeps THAT rule; only a
candidate the quota-off cut would have delivered is named as the quota's.
The budget step is lifted into `consume._pack` and used by both, so the
quota-off fate is decided by the code the run itself uses. This moves the
`rule` string a consumer reads for some withheld entries; no delivery, no
rank and no excerpt byte moves, and no committed payload in this repository
changed. The retrieval gate's row 3 goes **2 of 5 RED to 5 of 5 GREEN**.
### Added after the 0.10.1 notes were written (2026-09-20, untagged)
- **The payload says what of the question it reached** — a new top-level
`coverage` member carrying three lists: the terms the pre-pass read the
question as, the terms no concept in the bundle answers, and the terms no
delivered excerpt answers. Without it a reader holding eight excerpts cannot
tell a bundle that ANSWERED its question from one that merely ranked
something; the two payloads have the same shape. Documented as SS 8 point 7
of `docs/consumption-contract.md`, and the generated consumption skill is
told to read it.
- **Facts, and no verdict, which is a measurement rather than caution.** Two
readings were built and both falsified over **81 questions** (16 synthetic,
65 across three real gold sets, 2026-09-20): the share of a question's
terms a delivered excerpt answers separates the synthetic controls at 0.33
against 0.50 and then REVERSES on real data, where covered questions run
down to 0.27 while one genuinely uncovered question sits at 0.71; and the
share of a bundle tying the best lexical match is ~0.00 for every question
in a large bundle, covered or not. Question style dominates the first and
corpus size the second, so a pre-pass emitting a verdict would assert
across corpora what was measured on one.
- **Contract change, and the cost to a consumer is a re-run.** Every payload
grows the member; the checker does not read it, so a third-party pre-pass
that omits it stays conformant. The SS 7.4 known-positive moves with the
document it is measured on (14 721 / 375 → 16 389 / 417).
- **The retrieval gate is measurable where it was assertable**
(`tools/okf_retrieval_gate.py`, not shipped in the wheel):
- Row 8 prints the identity of every bundle it measured — path,
`bundle_id` and content ref — beside the set's sha256. Measured the same
day: two wiki bundles carrying the one `bundle_id` `claude-code-wiki` at
different refs score **6 of 20** and **5 of 20** on the same pinned set.
- `REAL_SET_PINS` states what each of the three real sets IS — questions,
fasit entries, controls and sha256 — so a self-written file in the right
shape is refused instead of reading `1 of 1 | 3 of 3 | GREEN`.
- Row 5 reads the hold-out threshold as a number in [0, 1] and RUNS the
registered set against the registered bundle, printing
`answered of asked = share against threshold`. `bool(threshold)` was the
whole check, so `report-only; any number is acceptable for v1` passed it.
- Row 4's marking reads the payload's `coverage`: `UNANSWERED_BAR = 2/3`,
swept and collapsing at both ends (at 0.50 eleven real covered questions
are marked; at 0.70 the row falls to 5 of 6). The margin is thin — 0.6087
against 0.6667 — and what it does not catch is published with it.
## [0.10.0] — 2026-09-17
### Added

53
CLAUDE.md
View file

@ -294,6 +294,59 @@ composition `process_inbox(segmentations=..., gate=inbox_gate)` now has a test
before this, `grep -rl inbox_gate tests/` gave **1** file with **0** occurrences
of `segment`, which is how the defect survived.
**AND THAT GATE WAS SILENT ABOUT WHAT IT DROPPED, UNTIL v1 (2026-09-20).**
Naming the gate in `log.md` said which screen ran; it did not say what the
screen took. Measured on an official documentation corpus of 594 sources at the
shipped default: `guard-trusted-source` refused **17 of 594 sources OUTRIGHT**
-- `fail_secure` 3, `quarantine_review` 14 -- and **16 of those sat among 197
documentation pages** (8.1 %), the pages on hooks, skills, permissions, errors,
env-vars and authentication among them. Rebuilt with `--gate none`, **17 of 17**
went through untouched, so the refusal is the GATE and not the readers: official
documentation naturally carries commands and instruction-shaped text, and the
guard reads that as something to hold for review. The summary printed `` `fail_secure`:
3/594 `` and nothing else, so **three of the four facts a reader needs were
absent**: the COUNT of documents the gate dropped (the existing `rejected
(coded)` line SUMS gate refusals and extraction failures -- two failures with
two different remedies, which is why `gate_refused` is its own column on
`CorpusReport`), the NAMES, and the way out. `corpus.REFUSED_HEADING` now opens
a section carrying all four, directly under the denominator, and
`refusal_banner()` says the same thing in one line on stderr because `okf build
> report.txt` is ordinary; both read the SAME field, so the loud line and the
summary cannot come to disagree about a number. The per-name list is capped at
`REFUSED_NAME_CAP` = 10 and `log.md` gains one bullet naming **every** refused
document, uncapped -- a cap is only honest if the rest is somewhere a reader can
go, and §9 already holds the one fact about a run the bundle cannot otherwise
recover. **THE EXIT CODE DOES NOT MOVE**: the build is valid, every refusal is
coded, the conservation identity holds, and the bundle is a true record of what
the gate allowed; what was wrong was the silence, not the status. A run the gate
refused nothing from is byte-identical in both places -- the known-negative in
`tests/test_gate_refusal_is_loud.py`, and the reason this cannot have moved a
byte measured here: **0 of the bundles this repository ships** was built with a
gate refusal. The gate's SEMANTICS are untouched and the guard is not touched
from here (another repo, another release, its own security assessment); that a
trusted-source tier should CARRY a `quarantine_review` document marked rather
than throw it is a candidate for the round after v1, not work.
**v1.0.0 (2026-09-20) IS THE SCOPE THIS TOOL IS FINISHED AT.** Operator
20.09.2026: «Vi må lukke OKF-jobben så snart vi praktisk kan. Vi gjør bare det
vi MÅ for at den skal være på v1. 3D visualisering er ikke med der.» It adds NO
capability over `v0.10.1`; what it adds is that the tool says what it does not
do -- the refusal section above, and a `Known limitations` section on the front
page carrying only numbers already measured (the gate's refusals and the way
out; the absent ceiling on what ONE RUN pays for images, where each decode link
is bounded at `MAX_FILTER_DECODE_BYTES` = 512 MiB but a 70 KB PDF with 16 images
each under the declared limit reached **851 MB peak RSS** and `RLIMIT_AS` is not
enforceable on Darwin; the three gates of this repository that are RED today --
retrieval **5, 7, 8, 9**, MCP **2**, accounting **2, 3, 6**, all three re-run on
the closing commit; what the content accounting does not count; and that nothing
here draws a bundle). The two `pip install` lines under "Install in detail" now
install `[extract]` -- the first screen did and those two did not, so the two
recipes produced different installations and the detailed one reported
`extractor_extra_missing`. After this tag the library is touched for defects
found in its OWN use; the next planned round is Google OKF v0.3, triggered by
v0.3 shipping and not by a date. Destination, frames and the candidate list live
in `PLAN.md` (local-only: `origin` is the public `open/` repo).
**A FENCED CODE BLOCK DECLARES NO STRUCTURE (F2, 2026-09-15).** The proposer
read every line with the same grammars, so `# Use the opus[1m] alias` inside a
```` ```bash ```` fence became a level-1 ATX heading. Two effects and the

79
README.md
View file

@ -10,7 +10,7 @@ types](#supported-file-types) lists each one with the evidence behind it.
Python 3.10+ and [uv](https://docs.astral.sh/uv/). One line:
```sh
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.1"
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v1.0.0"
```
## Use it
@ -36,6 +36,64 @@ Code](#consume-in-claude-code) for the same thing in steps and with several
bundles at once, [Build](#build) for the flags, [Requirements](#requirements)
for the pip fallback and the guard pairing.
## Known limitations
Read this before pointing the tool at documents you depend on. Every number
here was measured; none of it is a plan.
- **The default gate refuses whole documents, and they are documents you may
want.** Measured 2026-09-20 on an official documentation corpus of 594
sources: `guard-trusted-source`, the shipped default, refused **17 of 594
sources outright** — `fail_secure` 3, `quarantine_review` 14 — and **16 of
those were 197 official documentation pages** (8.1 %), among them the pages
on hooks, skills, permissions, errors, env-vars and authentication. Not one
element of a refused document reaches the bundle. Rebuilt with `--gate none`,
all 17 went through untouched, so the refusal is the gate and not the
readers: a page of official documentation naturally carries commands and
instruction-shaped text, and the guard reads that as something to hold for
review. The build says so now — it names the count, the files, the codes and
that command — and exits 0, because the bundle is a true record of what the
gate allowed. **If you vouch for the source yourself, build with `--gate
none`;** the bundle then records that nothing was screened. The default was
chosen on one measurement over one pinned bundle's 453 concept bodies, which
is a thin denominator for a decision this consequential.
- **Nothing bounds what one run pays for images.** Each decode link is capped
(`MAX_FILTER_DECODE_BYTES`, 512 MiB) and an oversized picture is refused with
its own code, but the run as a whole has no ceiling: measured, a 70 KB PDF
carrying 16 images each under the declared limit reached **851 MB peak RSS**
and every picture was carried. A hard cap outside Python was measured and is
not available here — `resource.setrlimit(RLIMIT_AS)` raises on Darwin 26.6.2
and is not enforced — so the per-link budget is the whole bound.
`--no-assets` takes the image path out entirely.
- **Three of this repository's own gates are RED, and each red row is a stated
finding rather than a bug to be surprised by.** The retrieval gate is red on
rows 5, 7, 8 and 9, the MCP gate on row 2, and the content accounting's judge
on rows 2, 3 and 6. For a user that means: retrieval quality is measured but
not yet green on a held-out set (rows 5, 8), two mechanical mutants of the
ranking survive with 0 ranks and 0 deliveries moved (row 7), no gold set
exists for the K2 corpus (row 9), MCP anchors and concept ids are different
vocabularies so `okf_fetch` cannot be addressed with a set's anchor (row 2),
and the accounting still reports real losses on the reference corpus (rows 2,
3, 6). The rows and their numbers are under [Judge the
retrieval](#judge-the-retrieval-python3-toolsokf_retrieval_gatepy) and
[Serve a bundle over MCP](#serve-a-bundle-over-mcp-okf-mcp).
- **The content accounting counts the element classes its vocabulary names, and
no others.** `0 unaccounted` is a statement about those classes, not about the
document: a file whose suffix has no reader is accounted at file level only,
parts no vocabulary names (headers, footers, endnotes, comments, speaker
notes, cell formulas) are outside it, and an image in `xlsx`, `md`, `txt`,
`csv`, `json`, `odt` or `rtf` is unaccounted and therefore red. It is opt-in
(`--accounting PATH`) for that reason. The full list is under
[Build](#build).
- **A few rough edges, named rather than fixed.** `okf skill` requires `--out`
and exits 2 without it. Over MCP, `okf_describe` requires `bundle_id` where
`okf_ask` treats it as optional. One `okf_ask` answer measured over 300 KB on
a 2 002-concept bundle, because the payload's `withheld` list carries one
entry per concept that was not delivered — the payload itself was well inside
its budget.
- **There is no context graph and no visualisation.** Nothing in this package
draws a bundle.
## What this library is
Status: phases 1–3 are implemented. Phase 1 (spec-based ingestion) covers
@ -120,11 +178,11 @@ Neither this package nor the guard it depends on is on a package index yet, so
both install by direct reference. With uv, one command resolves both:
```sh
uv pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.1"
uv pip install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v1.0.0"
```
uv resolves the guard on its own, because it reads the `[tool.uv.sources]`
entry in the `pyproject.toml` **of the tag it is installing**, and `v0.10.1`
entry in the `pyproject.toml` **of the tag it is installing**, and `v1.0.0`
points that entry at `llm-ingestion-guard` `v1.4.0`. Use `uv tool install`
instead of `uv pip install` when you want the `okf` command on `PATH` without an
active virtualenv — that is the form the first screen shows.
@ -135,10 +193,10 @@ With plain pip, the transitive git dependency does not resolve on its own —
```sh
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.4.0"
pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.1"
pip install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v1.0.0"
```
The guard tag is paired to the okf tag, not to this branch. `v0.10.1` declares
The guard tag is paired to the okf tag, not to this branch. `v1.0.0` declares
`llm-ingestion-guard>=1.2,<2.0`, which `v1.4.0` satisfies; the pairing above is
read off that tag's own `[tool.uv.sources]`, not off this branch. Reading a pin
off `main` and installing it against an older okf tag is the one combination
@ -149,7 +207,14 @@ that fails.
These are not install lines. They record what each earlier tag was, so a reader
who meets one in an older document knows what they are looking at.
- `v0.10.1` — the current tag: the image path of `v0.10.0`, with the two
- `v1.0.0` — the current tag, and the version this tool is finished at. It
adds no capability to `v0.10.1`: a document the gate refuses whole is now
named in the run's own summary with its code and with the one command that
carries it anyway, and the front page states what this tool does not do.
Read [Known limitations](#known-limitations) before you depend on it. After
this tag the library is touched for defects found in its own use, and the
next round is Google OKF v0.3.
- `v0.10.1` — the image path of `v0.10.0`, with the two
defects an independent review found in it closed. A remote `<img src>` or
`xlink:href` is inert text with the address in one code span, never a live
markdown image link, and no longer loses the figure's caption. An image is
@ -1103,7 +1168,7 @@ not need this repository — the first line installs the command, the second
builds the bundle and writes a skill beside it, the third asks.
```sh
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.1"
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v1.0.0"
okf project ~/my-documents
claude
```

2
pyproject.toml
View file

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "llm-ingestion-okf"
version = "0.10.1"
version = "1.0.0"
description = "Shared OKF (Open Knowledge Format) ingestion library: spec-based connectors, bundle inbox, and external-bundle import, with security delegated to llm-ingestion-guard."
readme = "README.md"
license = "MIT"

2
src/llm_ingestion_okf/__init__.py
View file

@ -76,7 +76,7 @@ from .manifest import (
)
from .materialize import IngestResult, materialize_bundle
__version__ = "0.10.1"
__version__ = "1.0.0"
__all__ = [
"BlockedFile",

5
src/llm_ingestion_okf/cli.py
View file

@ -1155,6 +1155,11 @@ def main(argv: list[str] | None = None) -> int:
args.report.parent.mkdir(parents=True, exist_ok=True)
args.report.write_text(report.render(), encoding="utf-8", newline="")
print(report.render())
# Loud where a redirected stdout cannot hide it. `okf build > report.txt`
# is an ordinary thing to do, and a document the gate dropped is the one
# thing about the run a reader must not learn later, or never.
if report.gate_refused:
print(f"{CLI_ID}: {report.refusal_banner()}", file=sys.stderr)
if report.conservation_failed:
print(
f"{CLI_ID}: K1b FAILED - {report.identity()}. Unaccounted: "

88
src/llm_ingestion_okf/corpus.py
View file

@ -79,6 +79,16 @@ HARNESS_ID = "okf-corpus-run"
LOG_NAME = "log.md"
LOG_TITLE = "Corpus run history"
#: The heading of the summary section that names the documents the gate refused
#: WHOLE, in ONE place because the section, the stderr banner and the tests all
#: have to mean the same section.
REFUSED_HEADING = "## Documents the gate refused WHOLE"
#: How many refused documents the printed summary names before it stops and
#: points at `log.md`. A cap is not a licence to lose the rest: `log.md` carries
#: every name, uncapped, and the section says so on the line where it stops.
REFUSED_NAME_CAP = 10
#: The permissive stub. It approves EVERYTHING, and it is named rather than
#: anonymous because that is the whole lesson of F1: from the day `okf build`
@ -244,6 +254,14 @@ class CorpusReport:
carried_files: int = 0
#: The content accounting of this run, or None when it was not asked for.
accounting: Accounting | None = None
#: Every document the GATE refused whole, as (source file, disposition),
#: sorted. Its own column because `rejected` sums two different failures
#: with two different remedies: a document the extractor could not read is
#: a format this library does not handle, while a document the gate refused
#: is a screening decision the caller can overrule with `--gate none`.
#: Measured 2026-09-20 on a 594-file documentation corpus, where 17 sources
#: were refused whole and the summary named neither the count nor a name.
gate_refused: tuple[tuple[str, str], ...] = ()
@property
def merged(self) -> int:
@ -273,6 +291,65 @@ class CorpusReport:
f"{self.merged} + {self.carried_files} + {self.rejected} = {total}; N = {self.n}"
)
def refusal_section(self) -> list[str]:
"""The four facts a reader needs about a document the gate dropped.
How many (with the denominator), which ones, under which code, and the
one command that carries them anyway. Empty when the gate refused
nothing, so a clean run's summary is the summary it always was.
"""
if not self.gate_refused:
return []
counts: dict[str, int] = {}
for _, disposition in self.gate_refused:
counts[disposition] = counts.get(disposition, 0) + 1
by_code = ", ".join(f"`{code}`: {count}" for code, count in sorted(counts.items()))
lines = [
REFUSED_HEADING,
"",
f"The gate `{self.gate}` refused {len(self.gate_refused)} of {self.n} "
"document(s) outright: not one element of them reached the bundle. "
f"By code: {by_code}.",
"",
]
lines.extend(
f"- `{name}` -- `{disposition}`"
for name, disposition in self.gate_refused[:REFUSED_NAME_CAP]
)
remaining = len(self.gate_refused) - REFUSED_NAME_CAP
if remaining > 0:
lines.append(
f"- ... and {remaining} more, every one named in the bundle's `{LOG_NAME}`."
)
lines += [
"",
"If you trust these sources yourself -- your own folder, your own "
"documents -- build them with `--gate none`. It screens nothing, and "
"the bundle records that it screened nothing.",
"",
]
return lines
def refusal_banner(self) -> str:
"""The same fact in one line, for a run whose stdout went to a file.
Built from `refusal_section`'s own data rather than beside it, so the
loud line and the summary cannot come to disagree about a number.
"""
if not self.gate_refused:
return ""
shown = ", ".join(
f"{name} (`{disposition}`)" for name, disposition in self.gate_refused[:3]
)
remaining = len(self.gate_refused) - 3
if remaining > 0:
shown += f", and {remaining} more"
return (
f"the gate `{self.gate}` refused {len(self.gate_refused)} of {self.n} "
f"document(s) WHOLE: {shown}. Trust these sources? build with "
f"`--gate none`. Full list in the summary above and in {LOG_NAME}"
)
def render(self) -> str:
per_file = self.seconds_total / self.n if self.n else 0.0
lines = [
@ -280,6 +357,7 @@ class CorpusReport:
"",
f"N (denominator, the directory's file count) = {self.n}",
"",
*self.refusal_section(),
"## Three counts, never one",
"",
"The guard sits between extraction and persist, so a healthy persisted",
@ -364,6 +442,15 @@ class CorpusReport:
"Every persisted byte of this bundle passed it.",
self._assets_line(),
]
if self.gate_refused:
named = ", ".join(
f"`{name}` (`{disposition}`)" for name, disposition in self.gate_refused
)
lines.append(
f"* **Refused whole by the gate**: {len(self.gate_refused)} of {self.n} "
f"document(s), so not one element of them is in this bundle: {named}. "
"Rebuild with `--gate none` to carry sources you vouch for yourself."
)
if self.accounting is not None:
lines.extend(self.accounting.log_lines())
if self.unaccounted:
@ -533,6 +620,7 @@ def measure(
assets_found=len(result.assets) + len(result.assets_rejected),
carried_files=len(carried),
accounting=account_run(corpus, walked, result) if account else None,
gate_refused=tuple(sorted((item.source_file, item.disposition) for item in blocked)),
)

196
tests/test_gate_refusal_is_loud.py Normal file
View file

@ -0,0 +1,196 @@
"""A document the gate refuses WHOLE is named in the run's own summary.
Measured on 2026-09-20 against a real documentation corpus (594 source files,
197 official documentation pages) built with the shipped default gate: 17
sources -- 16 of the 197 pages -- were refused outright, `fail_secure` 3 and
`quarantine_review` 14, and the pages lost were hooks, skills, permissions,
errors, env-vars and authentication. The build exited 0 and said so like this:
- `fail_secure`: 3/594
- `quarantine_review`: 14/594
Three of the four facts a reader needs were absent. The code was there (c);
the COUNT of documents the gate refused whole was not (the `rejected (coded)`
line sums gate refusals and extraction failures into one number), the NAMES
were not (b), and the way out -- `--gate none` for a source you vouch for
yourself -- was not (d). So a bundle could lose the pages it exists for while
its summary read like a clean run.
The exit code deliberately does NOT move. The build is valid: every refusal is
coded, the conservation identity holds, and the bundle is a true record of what
the gate allowed. What was wrong was the silence, not the status.
The fixture is a two-document inbox, never the real corpus: `CARRIER` carries a
zero-width space inside a word and is measured `fail_secure` under BOTH guard
presets, and `BENIGN` clears both, so the run has a refusal AND a survivor and
cannot pass by refusing everything.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from llm_ingestion_okf import cli, corpus
BUNDLE_ID = "gate-loud-fixture"
OKF_VERSION = "0.2"
CARRIER = "# Kostnader\n\nEn merknad med et nullbredde​tegn i seg.\n"
BENIGN = "# Kostnader\n\nKvartalstall for plattformgruppen, uten funn.\n"
def _inbox(root: Path, documents: dict[str, str]) -> Path:
inbox = root / "inbox"
inbox.mkdir(parents=True, exist_ok=True)
for name, body in documents.items():
(inbox / name).write_text(body, encoding="utf-8", newline="")
return inbox
def _build(inbox: Path, bundle: Path, *extra: str) -> int:
return cli.main(
[
"build",
str(inbox),
"--bundle",
str(bundle),
"--bundle-id",
BUNDLE_ID,
"--okf-version",
OKF_VERSION,
*extra,
]
)
def _section(out: str) -> str:
"""Exactly the refusal section, bounded at the next heading.
Taken to the end of the report, the slice would also hold `## Rejection
codes`, and an assertion about how many names the section lists would count
that section's totals too.
"""
after = out.split(corpus.REFUSED_HEADING, 1)[1]
end = after.find("\n## ")
return after if end < 0 else after[:end]
def _run(
tmp_path: Path, documents: dict[str, str], capsys: pytest.CaptureFixture[str]
) -> tuple[int, str, str]:
pytest.importorskip("llm_ingestion_guard")
inbox = _inbox(tmp_path, documents)
code = _build(inbox, tmp_path / "bundle")
captured = capsys.readouterr()
return code, captured.out, captured.err
def test_the_refused_documents_are_counted_with_their_denominator(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""(a) how many documents the gate refused whole -- its own number.
`rejected (coded): 1/2` already existed and is not this number: it also
counts a file the extractor could not read, which is a different fact about
a different failure and points at a different remedy.
"""
code, out, _ = _run(tmp_path, {"carrier.md": CARRIER, "benign.md": BENIGN}, capsys)
assert code == 0, "a build whose gate refused one of two documents is still a valid build"
assert corpus.REFUSED_HEADING in out
assert "1 of 2" in out
def test_the_refused_documents_are_named(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""(b) the file names, and (c) the code beside each one."""
_, out, _ = _run(tmp_path, {"carrier.md": CARRIER, "benign.md": BENIGN}, capsys)
section = _section(out)
assert "carrier.md" in section
assert "fail_secure" in section
assert "benign.md" not in section, "a document that reached the bundle is not a refusal"
def test_the_way_out_is_stated_in_the_summary(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""(d) the exact command, not a hint that one exists."""
_, out, _ = _run(tmp_path, {"carrier.md": CARRIER, "benign.md": BENIGN}, capsys)
section = _section(out)
assert "--gate none" in section
assert "trust" in section.lower()
def test_a_run_the_gate_refused_nothing_from_says_nothing(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""The known-positive: the section must be earned, not unconditional.
Without this, every assertion above is satisfied by a constant string and
the test says nothing about whether a refusal was observed.
"""
code, out, err = _run(tmp_path, {"benign.md": BENIGN}, capsys)
assert code == 0
assert corpus.REFUSED_HEADING not in out
assert "--gate none" not in err
def test_the_refusal_also_reaches_stderr(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""Loud where a redirected stdout cannot hide it.
`okf build > report.txt` is an ordinary thing to do, and it puts the whole
summary in a file the reader opens later, if at all.
"""
_, _, err = _run(tmp_path, {"carrier.md": CARRIER, "benign.md": BENIGN}, capsys)
banner = [line for line in err.splitlines() if "refused" in line]
assert banner, "stderr must carry the refusal"
assert "carrier.md" in banner[0]
assert "--gate none" in banner[0]
def test_a_long_list_is_capped_and_points_at_the_whole_one(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""A capped list is only honest if the rest is somewhere a reader can go.
The bundle's own `log.md` is that somewhere: SPEC section 9 already holds
this run's denominator and gate name, and a document the gate dropped is
the same class of fact -- the one thing about the run the bundle cannot
otherwise recover.
"""
# Distinct titles, not thirteen copies: two documents reducing to one
# concept name hit the section 3 collision refusal BEFORE the gate, which
# would make this row green over a set the gate never saw.
documents = {
f"carrier-{index:02d}.md": CARRIER.replace("Kostnader", f"Kostnader {index:02d}")
for index in range(corpus.REFUSED_NAME_CAP + 3)
}
documents["benign.md"] = BENIGN
_, out, _ = _run(tmp_path, documents, capsys)
section = _section(out)
assert section.count("`fail_secure`") == corpus.REFUSED_NAME_CAP + 1, (
"the per-name list is capped; the count line carries the code totals"
)
assert "3 more" in section
assert corpus.LOG_NAME in section
log = (tmp_path / "bundle" / corpus.LOG_NAME).read_text(encoding="utf-8")
for name in sorted(documents):
if name != "benign.md":
assert name in log, "log.md names every refused document, uncapped"
def test_the_log_is_unchanged_when_the_gate_refused_nothing(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""Exposure, measured rather than argued: a clean run's bytes do not move.
Every bundle this repository ships was built without a gate refusal, so a
bullet written only when there is one cannot have moved a byte of them.
"""
_run(tmp_path, {"benign.md": BENIGN}, capsys)
log = (tmp_path / "bundle" / corpus.LOG_NAME).read_text(encoding="utf-8")
assert "refused" not in log.lower()

2
uv.lock generated
View file

@ -547,7 +547,7 @@ source = { git = "https://git.fromaitochitta.com/open/llm-ingestion-pipeline-sec
[[package]]
name = "llm-ingestion-okf"
version = "0.10.1"
version = "1.0.0"
source = { editable = "." }
dependencies = [
{ name = "llm-ingestion-guard" },