fix(gate,propose): okf build runs a real guard; a code fence declares no structure
Two defects reported from outside by claude-code-llm-wiki (order 20260915T202332Z-228694739), both reproduced against this tree before anything moved. F1 -- the packaged CLI never ran the guard. corpus.measure wired an unconditional approve-everything stub into process_inbox and 0 of 90 add_argument calls named a gate, so the one path most people use screened nothing while pyproject.toml made the guard a mandatory runtime dependency and the README recommended a composition the command line could not reach. --gate takes guard-trusted-source (default), guard-user-upload or none. corpus.resolve_gate is the one name->callable map, with the guard imported lazily so importing the package still does not pull it in; an unknown name RAISES rather than falling back, because a fallback reproduces the defect with an extra step. The gate's NAME goes into the section 9 log.md -- a stub is only dangerous when nothing downstream can see it -- and --gate none renders NOTHING WAS SCREENED. The default was chosen on a measurement: over the 453 concept bodies of the pinned reference bundle, PRESET_TRUSTED_SOURCE persists 453 of 453 and PRESET_USER_UPLOAD holds 1, costing that concept's whole source document. Neither tier waves anything through -- an invisible carrier and a CRITICAL finding fail secure at both. Door B's library default is UNCHANGED at PRESET_USER_UPLOAD: an inbox drop is an untrusted upload, an operator pointing this command at their own folder is not. The second tier ships as guard_adapter.inbox_gate_trusted_source, the three-line adapter that module's docstring already described, never a preset parameter. process_inbox(segmentations=..., gate=inbox_gate) now has a test. Before this, `grep -rl inbox_gate tests/` gave 1 file with 0 occurrences of `segment` -- the recommended composition was untested, which is how the defect survived. F2 -- a fenced code block declared structure. `# Use the opus[1m] alias` inside a ```bash fence became a level-1 ATX heading: the document was refused entirely where the line carried [ or ] (5 of 191 pages of the reporter's corpus), and the concept TITLE came from somebody's shell session on 62 of 191 (32.5 %). The fix is in the proposer and never in Door B's title rule -- that rule is right, and a heading that was never a heading is what has to stop being proposed. propose.fenced_lines is computed once per text and no rule reads a fenced line, including Arm D's outline RUN, which selects from the whole line list. Backtick and tilde fences, three leading spaces, a closing fence at least as long as its opener, and no backtick in a backtick fence's info string -- that last one keeps a line holding only `okf build` from silencing a document. MEASURED ON THE BYTES, and this is the number that decides: the 43-document reference corpus built atb6da09c(from git archive, never the editable tree) and rebuilt at the shipped defaults differ in log.md alone, by the one added bullet. 865 concept files on both sides, every concept byte-identical. Found by that control and NOT caused by this work: the pinned artifact K2-bundle-default-20260912 was written 2026-09-09, two days beforeed0418fchanged title: quoting, so it differs from what HEAD produces on 42 concept files. test_default_bundle_pin stays green because it pins the count and the hit@8 ranks, not the bytes. Re-pinning is the operator's call. Suite 1896 passed / 1 skipped (+27 from 1869). ruff, ruff format and mypy --strict clean. No version bump, no tag, no push. Report: docs/2026-09-15-f1-f2-gaten-og-kodefencen.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
b6da09cc97
commit
332961a19c
11 changed files with 1194 additions and 10 deletions
|
|
@ -74,7 +74,14 @@ from collections.abc import Mapping, Sequence
|
|||
from functools import partial
|
||||
from pathlib import Path
|
||||
|
||||
from .corpus import LOG_NAME, CorpusReport, load_plans, measure
|
||||
from .corpus import (
|
||||
GATE_GUARD_TRUSTED_SOURCE,
|
||||
GATE_NAMES,
|
||||
LOG_NAME,
|
||||
CorpusReport,
|
||||
load_plans,
|
||||
measure,
|
||||
)
|
||||
from .errors import IngestError
|
||||
from .extract import declared_identity
|
||||
from .inbox import validate_concept_frontmatter, walk_inbox
|
||||
|
|
@ -198,6 +205,32 @@ DEFAULT_CLOSE_SPAN_GAPS = True
|
|||
#: every consumer and not to this one. Turning it on is its own decision with
|
||||
#: its own denominator, and this line is not it.
|
||||
DEFAULT_SHELL_PARENT = False
|
||||
|
||||
#: `okf build`'s persist gate, by name. Set by the operator 2026-09-15, on a
|
||||
#: measurement and not on caution.
|
||||
#:
|
||||
#: Until then `corpus.measure` wired an unconditional approve-everything stub
|
||||
#: into every run and no argument anywhere named a gate, so the one path people
|
||||
#: use screened nothing while the guard sat in `pyproject.toml` as a mandatory
|
||||
#: runtime dependency and the README recommended a composition that was not
|
||||
#: reachable from the command line. Reported from outside (F1,
|
||||
#: `claude-code-llm-wiki`, 2026-09-15) and reproduced here before anything moved.
|
||||
#:
|
||||
#: `guard-trusted-source` rather than Door B's own `PRESET_USER_UPLOAD`, because
|
||||
#: the two are answering different questions -- an inbox drop is an untrusted
|
||||
#: upload, an operator pointing this command at their own folder is not -- and
|
||||
#: because the cost of each is measured rather than assumed: over the 453
|
||||
#: concept bodies of the pinned default bundle, the trusted tier returns the
|
||||
#: persist disposition on 453 of 453 while the upload tier holds 1, taking one
|
||||
#: of the 39 source documents out of the bundle. Neither tier waves anything
|
||||
#: through: an invisible carrier and a CRITICAL finding fail secure at both.
|
||||
#:
|
||||
#: This is the ONE default in this module that is not about segmentation, and
|
||||
#: it is the only one whose opt-out (`--gate none`) turns a screen OFF rather
|
||||
#: than a rule. The stub stays reachable by name because the corpus harness
|
||||
#: reproduces published numbers with it; what changed is that reaching it is an
|
||||
#: act, and `log.md` records which gate screened the bundle either way.
|
||||
DEFAULT_GATE = GATE_GUARD_TRUSTED_SOURCE
|
||||
#: Round 9. Clause 1 asked whether a title ENDED in a number, which is a
|
||||
#: question about the number rather than about the title: a drawing's dimension
|
||||
#: chain, a P&ID's schematic labels, a door schedule and a borehole log's
|
||||
|
|
@ -461,6 +494,7 @@ def build(
|
|||
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
|
||||
shell_parent: bool = DEFAULT_SHELL_PARENT,
|
||||
frontmatter: Mapping[str, str] | None = None,
|
||||
gate: str = DEFAULT_GATE,
|
||||
) -> CorpusReport:
|
||||
"""Folder in, bundle out. The whole command, minus argument parsing.
|
||||
|
||||
|
|
@ -500,6 +534,7 @@ def build(
|
|||
heading_reserve=reserve,
|
||||
ocr=ocr,
|
||||
concept_frontmatter_values=concept_values,
|
||||
gate=gate,
|
||||
)
|
||||
_write_log(bundle, report, profile=STRUCTURED_V1)
|
||||
return report
|
||||
|
|
@ -559,6 +594,7 @@ def build(
|
|||
heading_reserve=reserve,
|
||||
ocr=ocr,
|
||||
concept_frontmatter_values=concept_values,
|
||||
gate=gate,
|
||||
)
|
||||
_write_log(bundle, report, profile=SEGMENTED_OKF_V0_2)
|
||||
return report
|
||||
|
|
@ -861,6 +897,19 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
|||
"pre-2026-09-11 spans byte for byte"
|
||||
),
|
||||
)
|
||||
build_parser.add_argument(
|
||||
"--gate",
|
||||
choices=GATE_NAMES,
|
||||
default=DEFAULT_GATE,
|
||||
help=(
|
||||
"the persist gate every concept body passes before it is written. "
|
||||
"`guard-trusted-source` (default) and `guard-user-upload` are the "
|
||||
"real llm-ingestion-guard at its two presets; `none` is a "
|
||||
"permissive stub that approves EVERYTHING and screens nothing. "
|
||||
"The name is written into the bundle's log.md either way, so a "
|
||||
"consumer can tell a screened bundle from an unscreened one"
|
||||
),
|
||||
)
|
||||
build_parser.add_argument(
|
||||
"--shell-parent",
|
||||
action="store_true",
|
||||
|
|
@ -1034,6 +1083,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
ocr=args.ocr,
|
||||
pdf_outline=args.pdf_outline,
|
||||
shell_parent=args.shell_parent,
|
||||
gate=args.gate,
|
||||
frontmatter=frontmatter_from_flags(args.frontmatter or ()),
|
||||
)
|
||||
except (IngestError, OSError, ValueError) as exc:
|
||||
|
|
|
|||
|
|
@ -43,6 +43,7 @@ from pathlib import Path
|
|||
from .errors import IngestError
|
||||
from .extract import extract_text
|
||||
from .inbox import (
|
||||
Gate,
|
||||
GateDecision,
|
||||
InboxResult,
|
||||
process_inbox,
|
||||
|
|
@ -53,6 +54,10 @@ from .profiles import SEGMENTED_OKF_V0_2, STRUCTURED_V1, BundleProfile
|
|||
from .segmentation import SegmentationPlan, parse_segmentation_plan
|
||||
|
||||
__all__ = [
|
||||
"GATE_GUARD_TRUSTED_SOURCE",
|
||||
"GATE_GUARD_USER_UPLOAD",
|
||||
"GATE_NAMES",
|
||||
"GATE_NONE",
|
||||
"CorpusReport",
|
||||
"converter_identity",
|
||||
"load_plans",
|
||||
|
|
@ -60,6 +65,7 @@ __all__ = [
|
|||
"main",
|
||||
"measure",
|
||||
"replace",
|
||||
"resolve_gate",
|
||||
"unaccounted_names",
|
||||
]
|
||||
|
||||
|
|
@ -72,6 +78,69 @@ LOG_NAME = "log.md"
|
|||
LOG_TITLE = "Corpus run history"
|
||||
|
||||
|
||||
#: The permissive stub. It approves EVERYTHING, and it is named rather than
|
||||
#: anonymous because that is the whole lesson of F1: from the day `okf build`
|
||||
#: was packaged until 2026-09-15 this function was wired unconditionally into
|
||||
#: every run, so the only path anyone used screened nothing while the guard sat
|
||||
#: in `pyproject.toml` as a mandatory runtime dependency. A consumer had no way
|
||||
#: to see it. It survives because the corpus harness reproduces published
|
||||
#: numbers with it and a caller measuring segmentation alone has a legitimate
|
||||
#: reason to take the gate out of the picture -- but reaching it is now an act,
|
||||
#: and the bundle's own log says so.
|
||||
GATE_NONE = "none"
|
||||
#: `okf build`'s default, set by the operator 2026-09-15 on a measurement:
|
||||
#: over the 453 concept bodies of the pinned default bundle, this tier returns
|
||||
#: the persist disposition on 453 of 453, so the move costs the pinned artifact
|
||||
#: nothing while the scanner actually runs. Pointing the command at your own
|
||||
#: folder is the trusted-source case.
|
||||
GATE_GUARD_TRUSTED_SOURCE = "guard-trusted-source"
|
||||
#: Door B's own library default, exposed for a folder that IS an untrusted
|
||||
#: drop. Stricter, and the cost is measured rather than implied: on the same
|
||||
#: 453 bodies it holds 1, which takes one of the 39 source documents out.
|
||||
GATE_GUARD_USER_UPLOAD = "guard-user-upload"
|
||||
|
||||
#: Every gate a run may declare. A name outside this tuple is refused rather
|
||||
#: than resolved to the stub: falling back would reproduce F1 with an extra
|
||||
#: step -- the caller believes they asked for the guard, and the run approves
|
||||
#: everything.
|
||||
GATE_NAMES = (GATE_GUARD_TRUSTED_SOURCE, GATE_GUARD_USER_UPLOAD, GATE_NONE)
|
||||
|
||||
_GATE_DESCRIPTIONS = {
|
||||
GATE_GUARD_TRUSTED_SOURCE: "llm-ingestion-guard, PRESET_TRUSTED_SOURCE",
|
||||
GATE_GUARD_USER_UPLOAD: "llm-ingestion-guard, PRESET_USER_UPLOAD",
|
||||
GATE_NONE: "permissive stub -- NOTHING WAS SCREENED",
|
||||
}
|
||||
|
||||
|
||||
def _gate(text: str) -> GateDecision:
|
||||
return GateDecision(sanitized_text=text, disposition="warn")
|
||||
|
||||
|
||||
def resolve_gate(name: str) -> Gate:
|
||||
"""One place maps a gate NAME to the callable, and it is this one.
|
||||
|
||||
The guard is imported lazily, inside the branch that needs it, because
|
||||
`guard_adapter` is the only module in this library allowed to import the
|
||||
dependency and importing the package must not pull it in.
|
||||
"""
|
||||
if name == GATE_NONE:
|
||||
return _gate
|
||||
if name == GATE_GUARD_TRUSTED_SOURCE:
|
||||
from .guard_adapter import inbox_gate_trusted_source
|
||||
|
||||
return inbox_gate_trusted_source
|
||||
if name == GATE_GUARD_USER_UPLOAD:
|
||||
from .guard_adapter import inbox_gate
|
||||
|
||||
return inbox_gate
|
||||
raise IngestError(
|
||||
f"unknown gate {name!r}; one of {', '.join(GATE_NAMES)} -- a gate name that "
|
||||
"resolved to the permissive stub would approve everything while the caller "
|
||||
"believed they had asked for the guard",
|
||||
code="gate_invalid",
|
||||
)
|
||||
|
||||
|
||||
def is_degenerate(text: str) -> bool:
|
||||
"""Zero characters after stripping whitespace. The whole rule, in one line.
|
||||
|
||||
|
|
@ -127,6 +196,10 @@ class CorpusReport:
|
|||
converter_version: str
|
||||
codes: tuple[tuple[str, int], ...]
|
||||
unaccounted: tuple[str, ...]
|
||||
#: The gate that screened this run, by NAME. Defaulted LAST and to the
|
||||
#: stub so every existing construction site stays source-compatible; the
|
||||
#: two commands that build bundles both pass it explicitly.
|
||||
gate: str = GATE_NONE
|
||||
|
||||
@property
|
||||
def merged(self) -> int:
|
||||
|
|
@ -215,6 +288,9 @@ class CorpusReport:
|
|||
f"{self.merged} + {self.rejected} = {self.merged + self.rejected}; "
|
||||
f"N = {self.n}. The run exits non-zero when these differ.",
|
||||
f"* **Converter**: {self.converter_path}, version {self.converter_version}.",
|
||||
f"* **Gate**: {self.gate} "
|
||||
f"({_GATE_DESCRIPTIONS.get(self.gate, 'unrecognised gate name')}). "
|
||||
"Every persisted byte of this bundle passed it.",
|
||||
]
|
||||
if self.unaccounted:
|
||||
lines.append("* **Unaccounted**: " + ", ".join(self.unaccounted) + " — K1b FAILED.")
|
||||
|
|
@ -247,10 +323,6 @@ def load_plans(plans_dir: Path) -> dict[str, SegmentationPlan]:
|
|||
}
|
||||
|
||||
|
||||
def _gate(text: str) -> GateDecision:
|
||||
return GateDecision(sanitized_text=text, disposition="warn")
|
||||
|
||||
|
||||
def _split_merges(corpus: Path, result: InboxResult) -> tuple[int, int]:
|
||||
"""Merged files split into substantive and degenerate, by the stated rule.
|
||||
|
||||
|
|
@ -284,6 +356,7 @@ def measure(
|
|||
heading_reserve: Callable[[str], bool] | None = None,
|
||||
ocr: bool = False,
|
||||
concept_frontmatter_values: Mapping[str, str] | None = None,
|
||||
gate: str = GATE_NONE,
|
||||
) -> CorpusReport:
|
||||
"""Run the corpus through the door and count what happened.
|
||||
|
||||
|
|
@ -301,7 +374,7 @@ def measure(
|
|||
bundle,
|
||||
ingested_at,
|
||||
okf_type="reference",
|
||||
gate=_gate,
|
||||
gate=resolve_gate(gate),
|
||||
profile=profile,
|
||||
root_frontmatter_values=root_frontmatter_values,
|
||||
segmentations=plans,
|
||||
|
|
@ -342,6 +415,7 @@ def measure(
|
|||
converter_version=version,
|
||||
codes=tuple(sorted(counts.items())),
|
||||
unaccounted=unaccounted_names(dropped=dropped, merged=merged_names, coded=coded_names),
|
||||
gate=gate,
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -371,6 +445,17 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
|||
default=None,
|
||||
help="required with --plans-dir: what a consumer joins the bundle's concepts on",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--gate",
|
||||
choices=GATE_NAMES,
|
||||
default=GATE_GUARD_TRUSTED_SOURCE,
|
||||
help=(
|
||||
"the persist gate every concept body passes before it is written. The "
|
||||
"same name and the same default as `okf build --gate`, deliberately: the "
|
||||
"two paths are held byte-equal by a test, and two different defaults "
|
||||
"would make that equality depend on which command you ran"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--okf-version",
|
||||
default=None,
|
||||
|
|
@ -451,6 +536,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
plans=plans,
|
||||
profile=profile,
|
||||
root_frontmatter_values=root_values,
|
||||
gate=args.gate,
|
||||
)
|
||||
args.report.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.report.write_text(report.render(), encoding="utf-8", newline="")
|
||||
|
|
|
|||
|
|
@ -48,14 +48,14 @@ own output, not a second opinion on the guard's scan.
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
from llm_ingestion_guard import PRESET_USER_UPLOAD, screen_output
|
||||
from llm_ingestion_guard import PRESET_TRUSTED_SOURCE, PRESET_USER_UPLOAD, screen_output
|
||||
from llm_ingestion_guard import okf as guard_okf
|
||||
|
||||
from .errors import MaterializationError
|
||||
from .importer import BundleDecision, ImportDecision
|
||||
from .inbox import GateDecision
|
||||
|
||||
__all__ = ["import_gate", "inbox_gate"]
|
||||
__all__ = ["import_gate", "inbox_gate", "inbox_gate_trusted_source"]
|
||||
|
||||
|
||||
def inbox_gate(text: str) -> GateDecision:
|
||||
|
|
@ -73,6 +73,32 @@ def inbox_gate(text: str) -> GateDecision:
|
|||
)
|
||||
|
||||
|
||||
def inbox_gate_trusted_source(text: str) -> GateDecision:
|
||||
"""The same gate at `PRESET_TRUSTED_SOURCE` (a `Gate`).
|
||||
|
||||
This IS the three-line adapter the module docstring above describes, shipped
|
||||
rather than left for each caller to write, because `okf build` needs a
|
||||
second tier and the alternative was a CLI reaching past the seam into the
|
||||
guard's presets. Nothing else differs: the same one-directional translation,
|
||||
the same `screen_output` on the exact bytes that will be persisted.
|
||||
|
||||
The tier is a statement about WHERE the bytes came from, and the two doors
|
||||
are answering different questions. An inbox drop is an untrusted upload, so
|
||||
`inbox_gate` stays on `PRESET_USER_UPLOAD` and remains Door B's default.
|
||||
`okf build <folder>` is an operator pointing at their own directory, which
|
||||
is the trusted-source case. Nothing is waved through either way: an any-tier
|
||||
finding -- an invisible carrier, a CRITICAL hit -- fails secure at both
|
||||
tiers, measured against guard 1.4.0 on the fixtures in
|
||||
`tests/test_cli_gate.py`.
|
||||
"""
|
||||
decision = screen_output(text, PRESET_TRUSTED_SOURCE)
|
||||
return GateDecision(
|
||||
sanitized_text=text,
|
||||
disposition=str(decision.disposition.value),
|
||||
reasons=tuple(str(reason) for reason in decision.reasons),
|
||||
)
|
||||
|
||||
|
||||
def import_gate(bundle: dict[str, str], *, origin: str, channel: str) -> BundleDecision:
|
||||
"""Door C's persist gate over `okf.import_bundle` (an `ImportGate`).
|
||||
|
||||
|
|
|
|||
|
|
@ -237,6 +237,25 @@ STOP_WORDS = frozenset(
|
|||
# list items, quantities and page furniture. The gate is what makes the signal
|
||||
# a signal.
|
||||
_ATX = re.compile(r"^(?P<hashes>#{1,6})\s+(?P<title>\S.*?)\s*$")
|
||||
# A FENCED CODE BLOCK, and it is the one construct in markdown that says "the
|
||||
# lines inside me are not markdown". Every grammar above reads lines, so
|
||||
# without this a shell comment in a ```bash block was a level-1 heading --
|
||||
# reported from outside 2026-09-15 and reproduced before anything moved. Two
|
||||
# effects, and the smaller one is the visible one: the document is REFUSED
|
||||
# entirely when the line carries `[` or `]` (Door B validates a title fail-fast
|
||||
# and never repairs one, 5 of 191 pages of the reporter's corpus), and the
|
||||
# concept TITLE is silently taken from somebody's shell session everywhere else
|
||||
# (62 of 191, 32.5 %).
|
||||
#
|
||||
# Three details of CommonMark SS 4.5 are load-bearing here, and each one is a
|
||||
# way to get this wrong in the direction that REMOVES real boundaries:
|
||||
# up to three leading spaces still open a fence (a code block inside a list is
|
||||
# the ordinary case in technical documentation); a backtick fence's info string
|
||||
# may not contain a backtick (or a line holding only `okf build` opens a fence
|
||||
# and silences the rest of the document); and a closing fence must be at least
|
||||
# as long as the opening one (or a four-backtick block quoting a three-backtick
|
||||
# example closes on the quoted line).
|
||||
_FENCE = re.compile(r"^ {0,3}(?P<marker>`{3,}|~{3,})(?P<info>.*)$")
|
||||
_NUMBERED = re.compile(r"^(?P<number>\d+(?:\.\d+)+)\s+(?P<title>\S.*?)\s*$")
|
||||
_TABLE_ROW = re.compile(r"^\s*\|.*\|\s*$")
|
||||
# Arm E's grammar: a pandoc GRID-table rule line. The converter separates a grid
|
||||
|
|
@ -400,6 +419,49 @@ def outline_lines(text: str) -> list[tuple[int, int, str]]:
|
|||
return found
|
||||
|
||||
|
||||
def fenced_lines(lines: Sequence[str]) -> set[int]:
|
||||
"""Every line index inside a fenced code block, fence lines included.
|
||||
|
||||
A whole-text decision, computed before the scan for the same reason the
|
||||
outline run is: whether a line is inside a fence is a property of the lines
|
||||
ABOVE it, and every rule in `find_candidates` has to agree about it or two
|
||||
of them will read the same line differently.
|
||||
|
||||
The fence lines themselves are in the set. They are not candidates under
|
||||
any grammar here, and leaving them out would only invite a later rule to
|
||||
read them.
|
||||
|
||||
An UNCLOSED fence runs to the end of the document, which is CommonMark's
|
||||
own rule. The alternative -- treating an unterminated opener as ordinary
|
||||
text -- reads a truncated code listing as a document full of headings,
|
||||
which is this defect in its worst form rather than a repair of it.
|
||||
"""
|
||||
fenced: set[int] = set()
|
||||
marker: str | None = None
|
||||
for index, line in enumerate(lines):
|
||||
match = _FENCE.match(line)
|
||||
if marker is None:
|
||||
if match is None:
|
||||
continue
|
||||
opening = match.group("marker")
|
||||
if opening[0] == "`" and "`" in match.group("info"):
|
||||
continue
|
||||
marker = opening
|
||||
fenced.add(index)
|
||||
continue
|
||||
fenced.add(index)
|
||||
if match is None:
|
||||
continue
|
||||
closing = match.group("marker")
|
||||
if (
|
||||
closing[0] == marker[0]
|
||||
and len(closing) >= len(marker)
|
||||
and not match.group("info").strip()
|
||||
):
|
||||
marker = None
|
||||
return fenced
|
||||
|
||||
|
||||
def heading_reserve_applies(text: str, *, outline_run: int) -> bool:
|
||||
"""Whether this text needs a SECOND heading source, having no run of its own.
|
||||
|
||||
|
|
@ -779,6 +841,12 @@ def find_candidates(
|
|||
position += len(line)
|
||||
end_of_text = position
|
||||
|
||||
# The fenced lines, and NOTHING below reads one. A fence is the one
|
||||
# construct that declares its own contents not to be markdown, so every
|
||||
# grammar here has to agree about it -- including the two whole-text passes
|
||||
# below, which select from the line list rather than from the loop.
|
||||
fenced = fenced_lines(lines)
|
||||
|
||||
# Computed BEFORE the loop, and that is a correctness requirement rather
|
||||
# than a style choice: run selection is a whole-text decision (the LAST
|
||||
# maximal run wins, because a contents listing precedes the body it lists),
|
||||
|
|
@ -789,7 +857,11 @@ def find_candidates(
|
|||
# silently. Silent loss, not a raise: nothing would announce it.
|
||||
admitted: dict[int, str] = {}
|
||||
if outline_run > 0:
|
||||
runs = outline_runs(outline_lines(text), outline_run)
|
||||
# Filtered HERE and not at admission: run selection is a property of
|
||||
# the whole text, so a fenced install listing left in the input would
|
||||
# decide WHICH run wins and move a boundary in prose it never touched.
|
||||
unfenced = [entry for entry in outline_lines(text) if entry[0] not in fenced]
|
||||
runs = outline_runs(unfenced, outline_run)
|
||||
if runs:
|
||||
# LAST run, not longest and not first. Measured against both:
|
||||
# first-run opens segments inside the table of contents on 14/39
|
||||
|
|
@ -835,7 +907,15 @@ def find_candidates(
|
|||
|
||||
# D3's input, and the same whole-text reasoning as `admitted` above: a run
|
||||
# is a property of the line list, not of a line.
|
||||
sections = _sheet_section_rows(lines) if sheet_section_rows else {}
|
||||
sections = (
|
||||
{
|
||||
index: section
|
||||
for index, section in _sheet_section_rows(lines).items()
|
||||
if index not in fenced
|
||||
}
|
||||
if sheet_section_rows
|
||||
else {}
|
||||
)
|
||||
|
||||
marked: list[tuple[int, Candidate]] = []
|
||||
in_table = False
|
||||
|
|
@ -850,6 +930,14 @@ def find_candidates(
|
|||
open_block: int | None = None
|
||||
joined: set[int] = set()
|
||||
for index, line in enumerate(lines):
|
||||
if index in fenced:
|
||||
# The same state the fall-through below clears for any other line
|
||||
# that is not a table row: a fenced block interrupts a table, and
|
||||
# the fence's own lines must not reopen one.
|
||||
in_table = False
|
||||
rule_pending = False
|
||||
open_block = None
|
||||
continue
|
||||
if _TABLE_ROW.match(line):
|
||||
section = sections.get(index)
|
||||
if section is not None:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue