open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 5 Activity

fix(okf): a flow sequence admits '#' and ':' where YAML reads them as text

Browse source 
1.4.0 refused a flow-sequence scalar element carrying '#' or ':' anywhere.
okf writes `references: [...]` as a flow sequence of doc links, and an
entry may carry a #fragment or a scheme://. claude-code-llm-wiki measured
parse_frontmatter raising on 2 038 of 5 467 concepts of a bundle okf
produced (2026-09-17); reproduced here on the same bundle, 2 038 / 5 467
before and 0 / 5 467 after (0 / 4 466 and 0 / 5 530 on the two other
builds).

The two characters are now refused by position, where PyYAML 6.0.3 gives
them a meaning: '#' opening the element or following whitespace (a
comment), ':' opening or ending the element or preceding whitespace (a
mapping). Elsewhere they are scalar characters. The rest of the refused
set, the indicator rule and the no-mixing rule are unchanged.

Chosen over admitting the characters wholesale because `[a: b]` is a
mapping and `[a #b]` a comment to YAML; admitting those would hand the
consumer a value YAML does not read. The position checks are substring
tests, no regex: CPU time stays linear in element count and length
(100k -> 800k), about 10 % over 1.4.0, inside the existing 2 s bound.

Pinned upstream corpus unmoved at 6 / 53. 910 passed (+17), coverage
exit 0, redos-sweep exit 0, 45 LIMITATIONS entries.
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 12:44:09 +02:00
commit 639da03f8c
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
4 changed files with 119 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

36
CHANGELOG.md
View file

@ -5,6 +5,42 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Fixed — a flow sequence refused `#` and `:` where YAML reads them as text
`1.4.0` refused a flow-sequence scalar element carrying `#` or `:` anywhere.
okf writes `references: [...]` as a flow sequence of doc links, and an entry
may carry a `#fragment` or a `scheme://`. A consumer measured the cost
(claude-code-llm-wiki, 2026-09-17): `okf.parse_frontmatter` raised on **2 038
of 5 467** concepts of a bundle okf itself produced, so `okf.import_bundle`
returned a parse error instead of a verdict for 37 % of it. Reproduced here on
the same bundle before the fix (2 038 / 5 467); after it, **0 / 5 467** (and
0 / 4 466 and 0 / 5 530 on the consumer's two other builds).
The two characters are now refused by position, where YAML gives them a
meaning, ground-truthed against PyYAML 6.0.3 on every test row: a `#` that
opens the element or follows whitespace opens a comment, and a `:` that opens
or ends the element, or precedes whitespace, opens a mapping. Both stay
refused there. Elsewhere they are scalar characters, so `a#b`, `x://y#z`,
`claude-cli://open` and `https://e.com:8443/a` parse as the strings YAML reads.
The rest of the refused set (`{ } [ ] , " '`), the scalar-indicator rule and
the no-mixing rule are unchanged.
The pinned upstream corpus is unmoved at **6 / 53** (`_okf-upstream` @
`3fcbb9f`); its binding constraint is the top-level block mapping, not this
rule. The predicate stays linear: CPU time doubles with each doubling of
element count and element length (100k → 800k), about 10 % over `1.4.0`.
### Changed — documentation surfaces caught up with `1.3.0` and `1.4.0`
README, ADOPTION-BRIEF and BRIEF named `1.3.0` after the `1.4.0` tag. README
also said `resource` is not on the frontmatter allowlist and that a `sources`
block list of mappings is refused (both false since `1.3.0`), and described the
`resource` https-allowlist without saying it covers the top-level `resource`
only — `sources[].resource` is never URL-validated. LIMITATIONS listed
`tags: [a, b]` among the routes that fail (false since `1.4.0`). All corrected.
## [1.4.0] — 2026-09-08
### Added — `tags: [a, b, c]`, the one flow-sequence form SPEC §4.1 writes out

6
README.md
View file

@ -178,7 +178,11 @@ the dotted and inline-colon routes to a mapping still raise. A
*sequence* value has two carriers — the block list, and (as of `1.4.0`) the flow
sequence `tags: [a, b, c]`, which is SPEC §4.1's own skeleton — whose elements
are either all plain scalars or all flow mappings, never a mix. A scalar element
carrying any of `{ } [ ] : , " ' #` is refused rather than guessed at. See
carrying any of `{ } [ ] , " '` is refused rather than guessed at, and so is a
`#` or `:` where YAML reads it (a `#` opening the element or following
whitespace, a `:` opening or ending it or preceding whitespace); since `1.4.1`
`[/docs/a#anchor, vscode://x, https://e.com:8443/a]` parses, as it does in
YAML. See
[LIMITATIONS](docs/LIMITATIONS.md) for what that admits and what it still walls
off; **`resource` https-allowlist** (hard-rejects
`data:`/`javascript:`/`file:` before commit — a reject-gate, not defang — on the

37
src/llm_ingestion_guard/okf.py
View file

@ -81,7 +81,15 @@ _QUOTE_STARTS = frozenset("\"'")
# two splitters, the two collection openers and their closers, and the comment
# indicator. Refusing them is what lets the element be split on commas at the
# character level without a YAML quote state machine.
_FLOW_SCALAR_REFUSED = "{}[]:,\"'#"
#
# 1.4.1 - `:` and `#` are refused only where YAML gives them a meaning, not
# anywhere in the element: okf writes `references: [...]` with `#fragment`s and
# `scheme://` links, and refusing those raised on 2 038 of 5 467 concepts of a
# consumer's bundle (measured 2026-09-17, reproduced 2026-09-23). Ground truth,
# PyYAML 6.0.3: `#` opens a comment at the start of an element or after
# whitespace; `:` opens a mapping at the start of an element, at its end, or
# before whitespace. Elsewhere both are ordinary scalar characters.
_FLOW_SCALAR_REFUSED = "{}[],\"'"
# G3 - the one mapping form T2 can express (operator decision, 2026-08-21).
# Every key inside a mapping must be on this allowlist: the form is safe because
@ -937,7 +945,8 @@ def _parse_flow_sequence(value, parent_key=None):
docs/2026-09-07-limitations-44-maaling.md). The quoting and comma-splitting
problem that kept it refused is answered by refusing the characters that
create it rather than by parsing them: an element is a plain scalar only if
it is non-empty and carries none of ``{ } [ ] : , " ' #``, and it then
it is non-empty and carries none of ``{ } [ ] , " '``, carries ``#`` and
``:`` only where YAML reads them as scalar characters (1.4.1), and it then
passes the unchanged scalar indicator rule. Everything needing YAML
semantics to split or unquote correctly still raises.
@ -1019,9 +1028,17 @@ def _flow_sequence_scalar(element, value):
A quote would have to be stripped (this parser retains quotes, so it would
hand back a different value than YAML reads); a comma or a colon would have
to be split on; a bracket or a brace would open a second collection level,
which no carrier here admits; a ``#`` opens a comment. The unchanged
indicator rule then applies to what is left, exactly as it does to a
block-list item, so an anchor or an alias is no more a scalar here.
which no carrier here admits. The unchanged indicator rule then applies to
what is left, exactly as it does to a block-list item, so an anchor or an
alias is no more a scalar here.
``#`` and ``:`` are refused by position, where YAML reads them (1.4.1):
a ``#`` that opens the element or follows whitespace opens a comment, and a
``:`` that opens or ends the element, or precedes whitespace, opens a
mapping. ``a#b``, ``x://y`` and ``https://e.com:8443/a`` are plain scalars
to YAML and to this parser alike. The element has already been cut at the
next comma and stripped, so its end is the only place a ``:`` can meet the
separator.
"""
if not element:
raise OKFFrontmatterError(
@ -1033,5 +1050,15 @@ def _flow_sequence_scalar(element, value):
"a flow-sequence scalar admits plain scalars only, not %r: %r"
% (char, value)
)
if element[0] == "#" or " #" in element or "\t#" in element:
raise OKFFrontmatterError(
"a '#' at the start of a flow-sequence element or after whitespace "
"opens a comment: %r" % (value,)
)
if element[0] == ":" or element[-1] == ":" or ": " in element or ":\t" in element:
raise OKFFrontmatterError(
"a ':' at the start or end of a flow-sequence element, or before "
"whitespace, opens a mapping: %r" % (value,)
)
_reject_dangerous_value(element)
return element

46
tests/test_okf.py
View file

@ -1143,6 +1143,52 @@ def test_a_flow_sequence_scalar_element_must_be_a_plain_scalar(cid, fm):
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
# --- 1.4.1: `#` and `:` are refused where YAML gives them a meaning, not anywhere
# 1.4.0 refused both characters anywhere in an element. A consumer measured what
# that costs (claude-code-llm-wiki, 2026-09-17, reproduced here 2026-09-23 on
# the same bundle): okf writes `references: [...]` as a flow sequence of doc
# links, an entry may carry a `#fragment` or a `scheme://`, and 2 038 of 5 467
# concepts raised on it. YAML itself reads those as plain scalars. The boundary
# is where YAML puts it, ground-truthed against PyYAML 6.0.3 on each row below:
# `#` opens a comment only at the start of an element or after whitespace, and
# `:` opens a mapping only at the start of an element, at its end, or before
# whitespace. Everywhere else both are ordinary characters of the scalar.
@pytest.mark.parametrize("cid,fm,expected", [
("okf's own references line",
"references: [/docs/en/claude-platform-on-aws, /docs/en/amazon-bedrock#3-configure-claude-code]\n",
["/docs/en/claude-platform-on-aws", "/docs/en/amazon-bedrock#3-configure-claude-code"]),
("fragment mid-element", "tags: [a#b]\n", ["a#b"]),
("fragment at the end", "tags: [a#]\n", ["a#"]),
("custom URL scheme", "references: [claude-cli://open]\n", ["claude-cli://open"]),
("scheme then a second element", "references: [vscode://ext/x, y]\n", ["vscode://ext/x", "y"]),
("host and port", "references: [https://e.com:8443/a]\n", ["https://e.com:8443/a"]),
("colon between two characters", "tags: [a:b]\n", ["a:b"]),
("space before the colon, none after", "tags: [a :b]\n", ["a :b"]),
("scheme and fragment together", "references: [x://y#z]\n", ["x://y#z"]),
])
def test_a_flow_sequence_scalar_admits_hash_and_colon_where_yaml_does(cid, fm, expected):
key = fm.split(":", 1)[0]
assert parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")[0][key] == expected
@pytest.mark.parametrize("cid,fm", [
# Each of these is a ParserError or a mapping in PyYAML 6.0.3, so each stays
# refused: admitting it would hand back a value YAML does not read.
("hash opening an element", "tags: [#b]\n"),
("hash opening a second element", "tags: [a, #b]\n"),
("hash after a space", "tags: [a # b, c]\n"),
("hash after a tab", "tags: [a\t#b]\n"),
("colon ending an element", "tags: [a:]\n"),
("colon before the separator", "tags: [a:, b]\n"),
("colon opening an element", "tags: [:a]\n"),
("colon before a tab", "tags: [a:\tb]\n"),
])
def test_a_flow_sequence_scalar_still_refuses_hash_and_colon_where_yaml_reads_them(cid, fm):
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
@pytest.mark.parametrize("cid,fm", [
("scalar then mapping", "tags: [a, {b: c}]\n"),
("mapping then scalar", "sources: [{ id: a }, plain]\n"),