open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(build): name the documents the gate refused whole, and close as 1.0.0

Browse source 
Measured 2026-09-20 on an official documentation corpus of 594 sources built
with the shipped default gate `guard-trusted-source`: 17 sources were refused
OUTRIGHT -- `fail_secure` 3, `quarantine_review` 14 -- and 16 of them were
among 197 official documentation pages, the pages on hooks, skills,
permissions, errors, env-vars and authentication among them. The summary said
only `fail_secure`: 3/594. Three of the four facts a reader needs were missing:
the COUNT of documents the gate dropped (the existing `rejected (coded)` line
sums gate refusals and extraction failures, two failures with two different
remedies), the NAMES, and the way out. Rebuilt with `--gate none`, all 17 went
through untouched, so the refusal is the gate and not the readers.

`okf build` now prints a `Documents the gate refused WHOLE` section directly
under the denominator, carrying all four: the count with its denominator, the
names capped at ten with the rest in the bundle's `log.md`, the codes, and
`--gate none` for a source you vouch for yourself. The same fact goes to stderr
in one line, built from the same field, because `okf build > report.txt` is an
ordinary thing to do. `log.md` gains one bullet naming every refused document,
uncapped.

The exit code deliberately does not move. The build is valid -- every refusal
is coded, the conservation identity holds, and the bundle is a true record of
what the gate allowed. What was wrong was the silence.

A run the gate refused nothing from is byte-identical in both places, which is
the known-negative in the new suite: no bundle this repository ships was built
with a gate refusal, so this cannot have moved a byte measured here.

Also, and measuring nothing new:

- README gains `Known limitations` high up -- the gate's refusals and the way
  out, the absent ceiling on what one run pays for images (a 70 KB PDF with 16
  images under the declared limit reached 851 MB peak RSS; RLIMIT_AS is not
  enforceable on this platform, so the 512 MiB per-link budget is the whole
  bound), the three gates of this repository that are RED today (retrieval
  5/7/8/9, MCP 2, accounting 2/3/6 -- all three re-run on this commit), what
  the content accounting does not count, and the rough edges nothing is
  planned for.
- The two `pip install` lines under "Install in detail" install `[extract]`.
  The first screen does; those two did not, so the two recipes produced
  different installations and the detailed one reports `resolved converter
  path: unresolved (extractor_extra_missing)`.
- Version `1.0.0`, synced across pyproject, `__version__`, `uv.lock`, the four
  README install lines, the install prose, the current-tag entry and the
  CHANGELOG, where the two "after the 0.10.1 notes were written, untagged"
  sections are folded in. It adds no capability over `v0.10.1`; what it adds
  is that the tool says what it does not do.

Suite: 2325 passed, 2 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-20 21:44:49 +02:00
commit 404bed26e2
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
10 changed files with 512 additions and 69 deletions
Download patch file Download diff file Expand all files Collapse all files

53
CLAUDE.md
View file

@ -294,6 +294,59 @@ composition `process_inbox(segmentations=..., gate=inbox_gate)` now has a test
before this, `grep -rl inbox_gate tests/` gave **1** file with **0** occurrences
of `segment`, which is how the defect survived.
**AND THAT GATE WAS SILENT ABOUT WHAT IT DROPPED, UNTIL v1 (2026-09-20).**
Naming the gate in `log.md` said which screen ran; it did not say what the
screen took. Measured on an official documentation corpus of 594 sources at the
shipped default: `guard-trusted-source` refused **17 of 594 sources OUTRIGHT**
-- `fail_secure` 3, `quarantine_review` 14 -- and **16 of those sat among 197
documentation pages** (8.1 %), the pages on hooks, skills, permissions, errors,
env-vars and authentication among them. Rebuilt with `--gate none`, **17 of 17**
went through untouched, so the refusal is the GATE and not the readers: official
documentation naturally carries commands and instruction-shaped text, and the
guard reads that as something to hold for review. The summary printed `` `fail_secure`:
3/594 `` and nothing else, so **three of the four facts a reader needs were
absent**: the COUNT of documents the gate dropped (the existing `rejected
(coded)` line SUMS gate refusals and extraction failures -- two failures with
two different remedies, which is why `gate_refused` is its own column on
`CorpusReport`), the NAMES, and the way out. `corpus.REFUSED_HEADING` now opens
a section carrying all four, directly under the denominator, and
`refusal_banner()` says the same thing in one line on stderr because `okf build
> report.txt` is ordinary; both read the SAME field, so the loud line and the
summary cannot come to disagree about a number. The per-name list is capped at
`REFUSED_NAME_CAP` = 10 and `log.md` gains one bullet naming **every** refused
document, uncapped -- a cap is only honest if the rest is somewhere a reader can
go, and §9 already holds the one fact about a run the bundle cannot otherwise
recover. **THE EXIT CODE DOES NOT MOVE**: the build is valid, every refusal is
coded, the conservation identity holds, and the bundle is a true record of what
the gate allowed; what was wrong was the silence, not the status. A run the gate
refused nothing from is byte-identical in both places -- the known-negative in
`tests/test_gate_refusal_is_loud.py`, and the reason this cannot have moved a
byte measured here: **0 of the bundles this repository ships** was built with a
gate refusal. The gate's SEMANTICS are untouched and the guard is not touched
from here (another repo, another release, its own security assessment); that a
trusted-source tier should CARRY a `quarantine_review` document marked rather
than throw it is a candidate for the round after v1, not work.
**v1.0.0 (2026-09-20) IS THE SCOPE THIS TOOL IS FINISHED AT.** Operator
20.09.2026: «Vi må lukke OKF-jobben så snart vi praktisk kan. Vi gjør bare det
vi MÅ for at den skal være på v1. 3D visualisering er ikke med der.» It adds NO
capability over `v0.10.1`; what it adds is that the tool says what it does not
do -- the refusal section above, and a `Known limitations` section on the front
page carrying only numbers already measured (the gate's refusals and the way
out; the absent ceiling on what ONE RUN pays for images, where each decode link
is bounded at `MAX_FILTER_DECODE_BYTES` = 512 MiB but a 70 KB PDF with 16 images
each under the declared limit reached **851 MB peak RSS** and `RLIMIT_AS` is not
enforceable on Darwin; the three gates of this repository that are RED today --
retrieval **5, 7, 8, 9**, MCP **2**, accounting **2, 3, 6**, all three re-run on
the closing commit; what the content accounting does not count; and that nothing
here draws a bundle). The two `pip install` lines under "Install in detail" now
install `[extract]` -- the first screen did and those two did not, so the two
recipes produced different installations and the detailed one reported
`extractor_extra_missing`. After this tag the library is touched for defects
found in its OWN use; the next planned round is Google OKF v0.3, triggered by
v0.3 shipping and not by a date. Destination, frames and the candidate list live
in `PLAN.md` (local-only: `origin` is the public `open/` repo).
**A FENCED CODE BLOCK DECLARES NO STRUCTURE (F2, 2026-09-15).** The proposer
read every line with the same grammars, so `# Use the opus[1m] alias` inside a
```` ```bash ```` fence became a level-1 ATX heading. Two effects and the