open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(gate,propose): okf build runs a real guard; a code fence declares no structure

Browse source 
Two defects reported from outside by claude-code-llm-wiki (order
20260915T202332Z-228694739), both reproduced against this tree before
anything moved.

F1 -- the packaged CLI never ran the guard. corpus.measure wired an
unconditional approve-everything stub into process_inbox and 0 of 90
add_argument calls named a gate, so the one path most people use screened
nothing while pyproject.toml made the guard a mandatory runtime dependency
and the README recommended a composition the command line could not reach.

  --gate takes guard-trusted-source (default), guard-user-upload or none.
  corpus.resolve_gate is the one name->callable map, with the guard imported
  lazily so importing the package still does not pull it in; an unknown name
  RAISES rather than falling back, because a fallback reproduces the defect
  with an extra step. The gate's NAME goes into the section 9 log.md -- a
  stub is only dangerous when nothing downstream can see it -- and --gate
  none renders NOTHING WAS SCREENED.

  The default was chosen on a measurement: over the 453 concept bodies of
  the pinned reference bundle, PRESET_TRUSTED_SOURCE persists 453 of 453 and
  PRESET_USER_UPLOAD holds 1, costing that concept's whole source document.
  Neither tier waves anything through -- an invisible carrier and a CRITICAL
  finding fail secure at both. Door B's library default is UNCHANGED at
  PRESET_USER_UPLOAD: an inbox drop is an untrusted upload, an operator
  pointing this command at their own folder is not. The second tier ships as
  guard_adapter.inbox_gate_trusted_source, the three-line adapter that
  module's docstring already described, never a preset parameter.

  process_inbox(segmentations=..., gate=inbox_gate) now has a test. Before
  this, `grep -rl inbox_gate tests/` gave 1 file with 0 occurrences of
  `segment` -- the recommended composition was untested, which is how the
  defect survived.

F2 -- a fenced code block declared structure. `# Use the opus[1m] alias`
inside a ```bash fence became a level-1 ATX heading: the document was
refused entirely where the line carried [ or ] (5 of 191 pages of the
reporter's corpus), and the concept TITLE came from somebody's shell session
on 62 of 191 (32.5 %). The fix is in the proposer and never in Door B's
title rule -- that rule is right, and a heading that was never a heading is
what has to stop being proposed. propose.fenced_lines is computed once per
text and no rule reads a fenced line, including Arm D's outline RUN, which
selects from the whole line list. Backtick and tilde fences, three leading
spaces, a closing fence at least as long as its opener, and no backtick in a
backtick fence's info string -- that last one keeps a line holding only
`okf build` from silencing a document.

MEASURED ON THE BYTES, and this is the number that decides: the 43-document
reference corpus built at b6da09c (from git archive, never the editable
tree) and rebuilt at the shipped defaults differ in log.md alone, by the one
added bullet. 865 concept files on both sides, every concept byte-identical.

Found by that control and NOT caused by this work: the pinned artifact
K2-bundle-default-20260912 was written 2026-09-09, two days before ed0418f
changed title: quoting, so it differs from what HEAD produces on 42 concept
files. test_default_bundle_pin stays green because it pins the count and the
hit@8 ranks, not the bytes. Re-pinning is the operator's call.

Suite 1896 passed / 1 skipped (+27 from 1869). ruff, ruff format and mypy
--strict clean. No version bump, no tag, no push.

Report: docs/2026-09-15-f1-f2-gaten-og-kodefencen.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-16 00:19:50 +02:00
commit 332961a19c
11 changed files with 1194 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

52
src/llm_ingestion_okf/cli.py
View file

@ -74,7 +74,14 @@ from collections.abc import Mapping, Sequence
from functools import partial
from pathlib import Path
from .corpus import LOG_NAME, CorpusReport, load_plans, measure
from .corpus import (
GATE_GUARD_TRUSTED_SOURCE,
GATE_NAMES,
LOG_NAME,
CorpusReport,
load_plans,
measure,
)
from .errors import IngestError
from .extract import declared_identity
from .inbox import validate_concept_frontmatter, walk_inbox
@ -198,6 +205,32 @@ DEFAULT_CLOSE_SPAN_GAPS = True
#: every consumer and not to this one. Turning it on is its own decision with
#: its own denominator, and this line is not it.
DEFAULT_SHELL_PARENT = False
#: `okf build`'s persist gate, by name. Set by the operator 2026-09-15, on a
#: measurement and not on caution.
#:
#: Until then `corpus.measure` wired an unconditional approve-everything stub
#: into every run and no argument anywhere named a gate, so the one path people
#: use screened nothing while the guard sat in `pyproject.toml` as a mandatory
#: runtime dependency and the README recommended a composition that was not
#: reachable from the command line. Reported from outside (F1,
#: `claude-code-llm-wiki`, 2026-09-15) and reproduced here before anything moved.
#:
#: `guard-trusted-source` rather than Door B's own `PRESET_USER_UPLOAD`, because
#: the two are answering different questions -- an inbox drop is an untrusted
#: upload, an operator pointing this command at their own folder is not -- and
#: because the cost of each is measured rather than assumed: over the 453
#: concept bodies of the pinned default bundle, the trusted tier returns the
#: persist disposition on 453 of 453 while the upload tier holds 1, taking one
#: of the 39 source documents out of the bundle. Neither tier waves anything
#: through: an invisible carrier and a CRITICAL finding fail secure at both.
#:
#: This is the ONE default in this module that is not about segmentation, and
#: it is the only one whose opt-out (`--gate none`) turns a screen OFF rather
#: than a rule. The stub stays reachable by name because the corpus harness
#: reproduces published numbers with it; what changed is that reaching it is an
#: act, and `log.md` records which gate screened the bundle either way.
DEFAULT_GATE = GATE_GUARD_TRUSTED_SOURCE
#: Round 9. Clause 1 asked whether a title ENDED in a number, which is a
#: question about the number rather than about the title: a drawing's dimension
#: chain, a P&ID's schematic labels, a door schedule and a borehole log's
@ -461,6 +494,7 @@ def build(
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
shell_parent: bool = DEFAULT_SHELL_PARENT,
frontmatter: Mapping[str, str] | None = None,
gate: str = DEFAULT_GATE,
) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing.
@ -500,6 +534,7 @@ def build(
heading_reserve=reserve,
ocr=ocr,
concept_frontmatter_values=concept_values,
gate=gate,
)
_write_log(bundle, report, profile=STRUCTURED_V1)
return report
@ -559,6 +594,7 @@ def build(
heading_reserve=reserve,
ocr=ocr,
concept_frontmatter_values=concept_values,
gate=gate,
)
_write_log(bundle, report, profile=SEGMENTED_OKF_V0_2)
return report
@ -861,6 +897,19 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"pre-2026-09-11 spans byte for byte"
),
)
build_parser.add_argument(
"--gate",
choices=GATE_NAMES,
default=DEFAULT_GATE,
help=(
"the persist gate every concept body passes before it is written. "
"`guard-trusted-source` (default) and `guard-user-upload` are the "
"real llm-ingestion-guard at its two presets; `none` is a "
"permissive stub that approves EVERYTHING and screens nothing. "
"The name is written into the bundle's log.md either way, so a "
"consumer can tell a screened bundle from an unscreened one"
),
)
build_parser.add_argument(
"--shell-parent",
action="store_true",
@ -1034,6 +1083,7 @@ def main(argv: list[str] | None = None) -> int:
ocr=args.ocr,
pdf_outline=args.pdf_outline,
shell_parent=args.shell_parent,
gate=args.gate,
frontmatter=frontmatter_from_flags(args.frontmatter or ()),
)
except (IngestError, OSError, ValueError) as exc: