open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(gate,propose): okf build runs a real guard; a code fence declares no structure

Browse source 
Two defects reported from outside by claude-code-llm-wiki (order
20260915T202332Z-228694739), both reproduced against this tree before
anything moved.

F1 -- the packaged CLI never ran the guard. corpus.measure wired an
unconditional approve-everything stub into process_inbox and 0 of 90
add_argument calls named a gate, so the one path most people use screened
nothing while pyproject.toml made the guard a mandatory runtime dependency
and the README recommended a composition the command line could not reach.

  --gate takes guard-trusted-source (default), guard-user-upload or none.
  corpus.resolve_gate is the one name->callable map, with the guard imported
  lazily so importing the package still does not pull it in; an unknown name
  RAISES rather than falling back, because a fallback reproduces the defect
  with an extra step. The gate's NAME goes into the section 9 log.md -- a
  stub is only dangerous when nothing downstream can see it -- and --gate
  none renders NOTHING WAS SCREENED.

  The default was chosen on a measurement: over the 453 concept bodies of
  the pinned reference bundle, PRESET_TRUSTED_SOURCE persists 453 of 453 and
  PRESET_USER_UPLOAD holds 1, costing that concept's whole source document.
  Neither tier waves anything through -- an invisible carrier and a CRITICAL
  finding fail secure at both. Door B's library default is UNCHANGED at
  PRESET_USER_UPLOAD: an inbox drop is an untrusted upload, an operator
  pointing this command at their own folder is not. The second tier ships as
  guard_adapter.inbox_gate_trusted_source, the three-line adapter that
  module's docstring already described, never a preset parameter.

  process_inbox(segmentations=..., gate=inbox_gate) now has a test. Before
  this, `grep -rl inbox_gate tests/` gave 1 file with 0 occurrences of
  `segment` -- the recommended composition was untested, which is how the
  defect survived.

F2 -- a fenced code block declared structure. `# Use the opus[1m] alias`
inside a ```bash fence became a level-1 ATX heading: the document was
refused entirely where the line carried [ or ] (5 of 191 pages of the
reporter's corpus), and the concept TITLE came from somebody's shell session
on 62 of 191 (32.5 %). The fix is in the proposer and never in Door B's
title rule -- that rule is right, and a heading that was never a heading is
what has to stop being proposed. propose.fenced_lines is computed once per
text and no rule reads a fenced line, including Arm D's outline RUN, which
selects from the whole line list. Backtick and tilde fences, three leading
spaces, a closing fence at least as long as its opener, and no backtick in a
backtick fence's info string -- that last one keeps a line holding only
`okf build` from silencing a document.

MEASURED ON THE BYTES, and this is the number that decides: the 43-document
reference corpus built at b6da09c (from git archive, never the editable
tree) and rebuilt at the shipped defaults differ in log.md alone, by the one
added bullet. 865 concept files on both sides, every concept byte-identical.

Found by that control and NOT caused by this work: the pinned artifact
K2-bundle-default-20260912 was written 2026-09-09, two days before ed0418f
changed title: quoting, so it differs from what HEAD produces on 42 concept
files. test_default_bundle_pin stays green because it pins the count and the
hit@8 ranks, not the bytes. Re-pinning is the operator's call.

Suite 1896 passed / 1 skipped (+27 from 1869). ruff, ruff format and mypy
--strict clean. No version bump, no tag, no push.

Report: docs/2026-09-15-f1-f2-gaten-og-kodefencen.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-16 00:19:50 +02:00
commit 332961a19c
11 changed files with 1194 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

92
src/llm_ingestion_okf/propose.py
View file

@ -237,6 +237,25 @@ STOP_WORDS = frozenset(
# list items, quantities and page furniture. The gate is what makes the signal
# a signal.
_ATX = re.compile(r"^(?P<hashes>#{1,6})\s+(?P<title>\S.*?)\s*$")
# A FENCED CODE BLOCK, and it is the one construct in markdown that says "the
# lines inside me are not markdown". Every grammar above reads lines, so
# without this a shell comment in a ```bash block was a level-1 heading --
# reported from outside 2026-09-15 and reproduced before anything moved. Two
# effects, and the smaller one is the visible one: the document is REFUSED
# entirely when the line carries `[` or `]` (Door B validates a title fail-fast
# and never repairs one, 5 of 191 pages of the reporter's corpus), and the
# concept TITLE is silently taken from somebody's shell session everywhere else
# (62 of 191, 32.5 %).
#
# Three details of CommonMark SS 4.5 are load-bearing here, and each one is a
# way to get this wrong in the direction that REMOVES real boundaries:
# up to three leading spaces still open a fence (a code block inside a list is
# the ordinary case in technical documentation); a backtick fence's info string
# may not contain a backtick (or a line holding only `okf build` opens a fence
# and silences the rest of the document); and a closing fence must be at least
# as long as the opening one (or a four-backtick block quoting a three-backtick
# example closes on the quoted line).
_FENCE = re.compile(r"^ {0,3}(?P<marker>`{3,}|~{3,})(?P<info>.*)$")
_NUMBERED = re.compile(r"^(?P<number>\d+(?:\.\d+)+)\s+(?P<title>\S.*?)\s*$")
_TABLE_ROW = re.compile(r"^\s*\|.*\|\s*$")
# Arm E's grammar: a pandoc GRID-table rule line. The converter separates a grid
@ -400,6 +419,49 @@ def outline_lines(text: str) -> list[tuple[int, int, str]]:
return found
def fenced_lines(lines: Sequence[str]) -> set[int]:
"""Every line index inside a fenced code block, fence lines included.
A whole-text decision, computed before the scan for the same reason the
outline run is: whether a line is inside a fence is a property of the lines
ABOVE it, and every rule in `find_candidates` has to agree about it or two
of them will read the same line differently.
The fence lines themselves are in the set. They are not candidates under
any grammar here, and leaving them out would only invite a later rule to
read them.
An UNCLOSED fence runs to the end of the document, which is CommonMark's
own rule. The alternative -- treating an unterminated opener as ordinary
text -- reads a truncated code listing as a document full of headings,
which is this defect in its worst form rather than a repair of it.
"""
fenced: set[int] = set()
marker: str | None = None
for index, line in enumerate(lines):
match = _FENCE.match(line)
if marker is None:
if match is None:
continue
opening = match.group("marker")
if opening[0] == "`" and "`" in match.group("info"):
continue
marker = opening
fenced.add(index)
continue
fenced.add(index)
if match is None:
continue
closing = match.group("marker")
if (
closing[0] == marker[0]
and len(closing) >= len(marker)
and not match.group("info").strip()
):
marker = None
return fenced
def heading_reserve_applies(text: str, *, outline_run: int) -> bool:
"""Whether this text needs a SECOND heading source, having no run of its own.
@ -779,6 +841,12 @@ def find_candidates(
position += len(line)
end_of_text = position
# The fenced lines, and NOTHING below reads one. A fence is the one
# construct that declares its own contents not to be markdown, so every
# grammar here has to agree about it -- including the two whole-text passes
# below, which select from the line list rather than from the loop.
fenced = fenced_lines(lines)
# Computed BEFORE the loop, and that is a correctness requirement rather
# than a style choice: run selection is a whole-text decision (the LAST
# maximal run wins, because a contents listing precedes the body it lists),
@ -789,7 +857,11 @@ def find_candidates(
# silently. Silent loss, not a raise: nothing would announce it.
admitted: dict[int, str] = {}
if outline_run > 0:
runs = outline_runs(outline_lines(text), outline_run)
# Filtered HERE and not at admission: run selection is a property of
# the whole text, so a fenced install listing left in the input would
# decide WHICH run wins and move a boundary in prose it never touched.
unfenced = [entry for entry in outline_lines(text) if entry[0] not in fenced]
runs = outline_runs(unfenced, outline_run)
if runs:
# LAST run, not longest and not first. Measured against both:
# first-run opens segments inside the table of contents on 14/39
@ -835,7 +907,15 @@ def find_candidates(
# D3's input, and the same whole-text reasoning as `admitted` above: a run
# is a property of the line list, not of a line.
sections = _sheet_section_rows(lines) if sheet_section_rows else {}
sections = (
{
index: section
for index, section in _sheet_section_rows(lines).items()
if index not in fenced
}
if sheet_section_rows
else {}
)
marked: list[tuple[int, Candidate]] = []
in_table = False
@ -850,6 +930,14 @@ def find_candidates(
open_block: int | None = None
joined: set[int] = set()
for index, line in enumerate(lines):
if index in fenced:
# The same state the fall-through below clears for any other line
# that is not a table row: a fenced block interrupts a table, and
# the fence's own lines must not reopen one.
in_table = False
rule_pending = False
open_block = None
continue
if _TABLE_ROW.match(line):
section = sections.get(index)
if section is not None: